Object-based video segmentation using spatio-temporal energy

Segmentation of moving object in a video sequence is one of the most challenging problems in video processing due to the complexity of its motion. This paper presents an approach of object segmentation with spatio-temporal energy. First, according to the dissimilar characteristic of the intra-frame and inter-frame (spatial and temporal) information, a novel energy model, which is based on Markov random fields and residual fields, is proposed. Then, an improved regions growing method is used to segment and label the image in inter-frame, and an object mask is achieved by the proposed spatio-temporal energy model, after morphology filter is adopted to get the better results, moving object are extracted. Finally, while the occlusion emerges due to overlapping motion between both objects, the covered/uncovered object could be identified by both the spatial and temporal energy which is rapidly changed in respect to the previous frame. The experiment results show that the approach is effective.