Extracting Multimedia Semantics Based on Independent Modality Discovering and Fusion

Learning semantics from low-level features of multimedia learning resources enables high-level access to multimedia content. Considerable amount of researches have been focused on multi-modal analysis to detect multimedia semantics. However, two fundamental issues have not been adequately addressed. First, given a set of raw features extracted from multimedia sources, what are the best independent modalities? Second, once a set of modalities has been identified, how are they optimally fused to map to the high-level semantics? In this paper, we apply statistical and machine learning techniques to answer the two questions. ISOMAP combining with support vector clustering are used to discover independent modalities from raw features. Then maximum entropy method is applied to optimally fuse the individual modalities. Experiments show that the proposed method can learn multimedia semantics more efficiently than traditional methods