Consistent 3D Background Model Estimation from Multi-viewpoint Videos

We present an approach for estimating the 3D background model of a scene from a collection of synchronized videos. Unlike previous work, our method is fully automatic, does not require empty frames depicting just the background, and makes very mild assumptions about the foreground. The constraint on the cameras is that they should have sufficiently narrow baselines to enable multi-view stereo matching. Using the images and primarily the depth maps as inputs, our algorithm detects potential background pixels to generate initial per-camera background models, which are then fused to form the final, consistent 3D background model. We show results on diverse video sequences captured using different camera configurations. Despite the challenges posed by the input videos, in which some parts of the background are always occluded in the images, we are able to extract accurate models of the background that are effective in foreground segmentation. This would have been impossible using conventional background subtraction methods that operate on the frames of each camera separately. Moreover, fusion makes the per-camera background models consistent.