Video semantic mining and annotation

Speech signal, video caption text and video frame images are all key factors for a person to understand the video content. Through above observation, we bring forward a scheme which integrating continuous speech recognition, video caption text recognition and object recognition. The video is firstly segmented into a number of shots by shot detection. The object recognition results are also presented in the same way. The above three folds of texts are processed by part of speech and stemming and finally are represented by three bags of words. Only the noun and verb words are kept. The words are further depicted as a graph. The graph vertices stand for the words and the edges denote the semantic relation between two neighboring words. In the last step, we apply the dense sub graph finding method to mine the video semantic meaning. Experiments show that our video semantic mining method is efficient.