Hierarchical multi-feature fusion for multimodal data analysis

Multimedia data is usually represented with different low-level features, and different types of multimedia data, namely multimodal data, often coexist in many data sources. It is interesting and challenging to learn comprehensive semantics from multiple low-level features for multimodal data analysis. In this paper, we propose a new algorithm, namely hierarchical multi-feature fusion for multimodal data semantics understanding. Our approach explores intra-modality structural information derived from each type of feature, and further proposes transductive inter-modality fusion strategy, which analyzes canonical correlation between different modalities. Extensive experiments are conducted on collected multimodal database for data classification application. The experiment results show that the performance of our algorithm is remarkable and demonstrate its superiority over several existing algorithms.