Interactive video segmentation supported by multiple modalities, with an application to depth maps

In this paper we propose an interactive method for the segmentation of objects in video. We aim to exploit multiple modalities to reduce the dependency on color discrimination alone. Given an initial segmentation for the first and last frame of a video sequence, we aim to propagate the segmentation to the intermediate frames of the sequence. Video frames are first segmented into superpixels. The segmentation propagation is then regarded as a superpixels labeling problem. The problem is formulated as an energy minimization problem which can be solved efficiently. Higher-order energy terms are included to represent temporal constraints. Our proposed method is interactive, to ensure correct propagation and relabel incorrectly labeled superpixels. As a final step the initial segmentation boundaries are refined to obtain accurate object boundaries. We then exploit these object boundaries in an application for computing depth maps.