Realistic inverse lighting from a single 2D image of a face, taken under unknown and complex lighting

In this paper, we address a difficult inverse rendering problem with many unknowns: a single 2D input image of an unknown face in an unknown environment, taken under unknown conditions. First, the geometry and texture of the face are estimated from the input image, using a 3D Morphable Model. In a second step, considering the superposition principle for light, we estimate the light source intensities as optimized non-negative weights for a linear combination of a synthetic illumination cone for that face. Each image of the illumination cone is lighted by one directional light, considering non-lambertian reflectance and non-convex geometry. Modeling the lighting separately from the face model enhances the face modeling and analysis, provides information about the environment of the face, and facilitates realistic rendering of the face in novel pose and lighting.