Linear multimodal fusion in video concept analysis based on node equilibrium model

Multiple modalities such as color, texture, shape and motion need to be analyzed separately and fused together to get the comprehensive result in content-based video concept analysis. We propose a multimodal fusion method based on a mechanical node equilibrium model. It treats the scores ofmultiple modalities and the fused score as physical nodes. Between these nodes, we define correlations which are treated as forces to move the nodes to a new position. Finally, the whole node system will be at an equilibrium status which is regarded as the fusion result. Essentially, the proposed method is a linear fusion model with linear fusion equations. The correlations are optimized by an expectation maximum (EM) algorithm which is quite efficient needing only several iterations.