Joint Key-Frame Extraction and Object Segmentation for Content-Based Video Analysis

Key-frame extraction and object segmentation are usually implemented independently and separately due to the fact that they are on different semantic levels and involve different features. In this work, we propose a joint key-frame extraction and object segmentation method by constructing a unified feature space for both processes, where key-frame extraction is formulated as a feature selection process for object segmentation in the context of Gaussian mixture model (GMM)-based video modeling. Specifically, two divergence-based criteria are introduced for key-frame extraction. One recommends key-frame extraction that leads to the maximum pairwise interclass divergence between GMM components. The other aims at maximizing the marginal divergence that shows the intra-frame variation of the mean density. The proposed methods can extract representative key-frames for object segmentation, and some interesting characteristics of key-frames are also discussed. This work provides a unique paradigm for content-based video analysis