Retexturing in the Presence of Complex Illumination and Occlusions

We present a nonrigid registration technique that achieves spatial, photometric, and visibility accuracy. It lets us photo-realistically augment 3D deformable surfaces under complex illumination conditions and in spite of severe occlusions. There are many approaches that address some of these issues but very few that simultaneously handle all of them as we do. We use triangulated meshes to model the geometry and introduce explicit visibility maps as well as separate illumination parameters for each mesh vertex. We cast our registration problem in an expectation maximization framework that allows robust and fully automated operation. It provides explicit illumination and occlusion models that can be used for rendering purposes.