Combining content and context information fusion for video classification and retrieval

Content-Based Video Retrieval has been a challenging problem and its performance relies on the modeling and representation of the video data and the underlying similarity metric. Most existing metrics evaluate pairwise shot similarity based only on shot perceptual content, which is denoted as content-based similarity. In this study, our concern is to recognize and detect video events that are “semantically similar”. Thus, we extend the content-based similarity to measure the conceptual content of shots. Here, conceptual content refers to the dynamic semantic concept which reflects a “human-action” regardless the perceptual/visual appearance. In addition, we propose a new similarity metric to make use of the shot contexts in video clips collection. The context of a shot is built by constructing a vector with each dimension representing the content similarity between the shot and any shot in the video collection. The context similarity between two videos is obtained by computing the similarity between the corresponding context vectors using the vector similarity functions. Furthermore, a linear and nonlinear fusion schemes are introduced to compute the relative contributions of each similarity in the overall retrieval and classification process. Experimental results demonstrate that the use of the context similarity can significantly improve the retrieval performance.