Robust foreground segmentation from sparsely arranged multi-view cameras

In this paper, we propose robust segmentation of foreground objects, such as human regions from sparsely arranged multi-view cameras. This work is intended for an immersive telepresence system. The system can be realized by segmenting a conferee (i.e. the foreground) from captured video at each conference site and then synthesizing life-sized textures with the background of another space. However, segmentation is a very challenging problem where the background has a similar texture to the foreground or the illumination varies according to time changes. Actually, segmentation accuracies of conventional methods are not sufficient to realize the telepresence system. The proposed method achieves sufficient segmentation quality to realize the system by directly estimating the foreground regions in a three-dimensional space based on the object existence probability for an individual camera and the color similarity among multiple cameras. Experimental results showed the effectiveness of the proposed method regarding foreground segmentation accuracy. Furthermore, confirmation was made that the experience of motion parallax for head movement could be naturally realized.