Robust foreground segmentation for GPU architecture in an immersive 3D videoconferencing system

Current telepresence systems, while being a great step forward in videoconferencing, still have important points to improve in what eye-contact, gaze and gesture awareness concerns. Many-to-many communications are going to greatly benefit from mature auto-stereoscopic 3D technology; allowing people to engage more natural remote meetings, with proper eye-contact and better spatiality feeling. For this purpose, proper real-time multi-perspective 3D video capture is necessary (often based on one or more View+Depth data sets). Given current state of the art, some sort of foreground segmentation is often necessary at the acquisition in order to generate 3D depth maps with hight enough resolution and accurate object boundaries. For this, one needs flicker-less foreground segmentations, accurate to borders, resilient to noise and foreground shade changes, and able to operate in real-time on performing architectures such as GPGPUs. This paper introduces a robust Foreground Segmentation approach used within the experimental immersive 3D Telepresence system from EU-FP7 3DPresence project. The proposed algorithm is based on a costs minimization using Hierarchical Believe Propagation and outliers reduction by regularization on oversegmented regions. The iterative nature of the approach makes it scalable in complexity, allowing it to increase accuracy and picture size capacity as GPGPUs become faster. In this work, particular care in the design of foreground and background cost models has also been taken in order to overcome limitations of previous work proposed in the literature.