A new multiview spacetime-consistent depth recovery framework for free viewpoint video rendering

In this paper, we present a new approach for recovering spacetime-consistent depth maps from multiple video sequences captured by stationary, synchronized and calibrated cameras for depth based free viewpoint video rendering. Our two-pass approach is generalized from the recently proposed region-tree based binocular stereo matching method. In each pass, to enforce temporal consistency between successive depth maps, the traditional region-tree is extended into a temporal one by including connections to “temporal neighbor regions” in previous video frames, which are identified using estimated optical flow information. For enforcing spatial consistency, multi-view geometric constraints are used to identify inconsistencies between depth maps among different views which are captured in an inconsistency map for each view. Iterative optimizations are performed to progressively correct inconsistencies through inconsistency maps based depth hypotheses pruning and visibility reasoning. Furthermore, the background depth and color information is generated from the results of the first pass and is used in the second pass to enforce sequence-wise temporal consistency and to aid in identifying and correcting spatial inconsistencies. The extensive experimental evaluations have shown that our proposed approach is very effective in producing spatially and temporally consistent depth maps.