Spatio-Temporal Registration Techniques for Relightable 3D Video

By jointly applying a model-based marker-less motion capture approach and multi-view texture generation 3D videos of human actors can be reconstructed from multi-view video streams. If the input data were recorded under calibrated lighting, the texture information can also be used to measure time-varying surface reflectance. This way, 3D videos can be realistically displayed under novel lighting conditions. Reflectance estimation is only feasible if the multi-view texture-to-surface registration is consistent over time. In this paper, we propose two image-based warping methods that compensate registration errors due to inaccurate model geometry and shifting of apparel over the body.