Fusion methods for multi-modal indexing of web data

Effective indexing of multimedia documents requires a multimodal approach in which either the most appropriate modality is selected or different modalities are used in a collaborative fashion. A collaborative pattern is a model of combination between media that defines how and when to combine information coming from different media sources. Fusing information coming from different media seems a natural way to handle multimedia content. We focus on describing fusion strategies where the task is achieved through the use of different modalities. We browse through the literature looking at various state of the art multi-modal fusion techniques varying from naive combination of modalities to more complex methods of machine learning and discuss various issues faced with fusing several modalities having different properties in the context of semantic indexing.