Recovery and tracking of continuous 3D surfaces from stereo data using a deformable dual-mesh

We propose a novel method for continuous 3D depth recovery and tracking using calibrated stereo. The method integrates stereo correspondence, surface reconstruction and tracking by using a new single deformable dual mesh optimization, resulting in simplicity, robustness and efficiency. In order to combine stereo correspondence and structure recovery, the method introduces an external energy function defined for a 3D volume based on cross-correlation between the stereo pairs. The internal energy functional of the deformable dual mesh imposes smoothness on the surfaces and it serves as a communication tool between the two meshes. Under the forces produced by the energy terms, the dual mesh deforms to recover and track the 3D surface. The newly introduced dual-mesh model, which is one of the main contributions of this paper, makes the system robust against local minima and yet it is efficient. A coarse-to-fine minimization approach makes the system even more efficient. Tracking is achieved by using the recovered surface as an initial position for the next time frame. Although the system can effectively utilize initial surface positions and disparity data, they are not needed for a successful operation, which makes this system applicable to a wide range of areas. We present the results of a number of experiments on stereo human face and cloud images, which proves that our new method is very effective