A multi-modal video analysis system

In this paper, we present a system for Chinese news program management based on cross media video analysis. Audio, caption text and video frames are all important for a person to understand the meaning of the video. Given these facts, we devised a system integrating continuous Chinese speech recognition (ASR), video caption text recognition (VOCR) and object/scene recognition (OR). The news program is firstly segmented to a serial of segments by anchor person detection. Then the ASR and VOCR recognition results are treated as two paragraphs of text, and we translate them to two bags of words to represent the original recognition results. By analysing the correspondance of the words in ASR result and VOCR result, we can get a trusted set of words to depict the video content of a segment of news program. In the last step, we implement the object/scene classification based on the keyframes analysis aided by the above recognition words. Experiments show that our news management system is efficient.