Semi-automatic motion based segmentation using long term motion trajectories

Semi-automated object segmentation is an important step in the cinema post-production workflow. We propose a dense motion based segmentation process that employs sparse feature based trajectories estimated across a long sequence of frames, articulated with a Bayesian framework. The algorithm first classifies the sparse trajectories into sparsely defined objects. Then the sparse object trajectories together with motion model side information are used to generate a dense object segmentation of each video frame. Unlike previous work, we do not use the sparse trajectories only to propose motion models, but instead use their position and motion throughout the sequence as part of the classification of pixels in the second step. Furthermore, we introduce novel colour and motion priors that employ the sparse trajectories to make explicit the spatiotemporal smoothness constraints important for long term motion segmentation.