A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces

Most of the previous work on template-based deformable 3D surface reconstruction using a single view is feature-based. We propose a pixel-based formulation in a variational framework; the unknown is the surface function. The color discrepancy between the template and the deformed images is formalized as a functional of the surface function. The main difficulty in such a formulation arises when the surface self-occludes which induces discontinuities in the discrepancy measure at the self-occlusion boundary. Based on previous work on 3D rigid surface reconstruction, we rigorously formalize the visibility as a continuous functional of the surface function. It is derived in the template for visible/self-occluded regions in the deformed image. The gradient of the color discrepancy is computed with respect to the surface function. The minimization smoothly updates the surface function to fit the self-occlusion boundary. Gradient descent is initialized from feature-based 3D reconstruction. Our experimental results on simulated and real data show that during the minimization of the color discrepancy, the self-occlusion boundary of the reconstructed surface moves to its correct location in the image. We show quantitatively that in the template image, the accuracy of visible/self-occluded areas is improved to a significant extent.