3D structure from video streams with partially overlapping images

The majority of methods available to recover 3D structure from video assume that a set of feature points are tracked across a large number of frames. This is not always possible in real videos because the images overlap only partially, due to occlusion and limited field of view. The paper describes a new method to recover 3D structures from videos with partially overlapping views. The well known factorization method recovers 3D rigid structures by factoring an observation matrix that collects the trajectories of feature points (see Tomasi, C. and Kanade, T., Int. J. of Computer Vision, vol.9, no.2, 1992). We extend this method to the more challenging scenario of observing incomplete trajectories. In this way, we accommodate not only features that disappear, but also features that, although not visible in the first image, become available later. Under this scenario, the observation matrix has missing entries. We develop three new algorithms to factor out matrices with missing data. Experiments with synthetic data and real video images demonstrate the viability of our approach to recover 3D structure.