A new audiovisual emotion recognition system using entropy-estimation-based multimodal information fusion

We present a novel audiovisual emotion recognition solution using multimodal information fusion based on entropy estimation. Considering the limitations of existing methods, we propose a new dual-level fusion framework which consists of feature level fusion module based on kernel entropy component analysis and score level fusion module based on maximum correntropy criterion. In our system, audio and visual channels are utilized to detect and classify emotional states for intelligent human computer interfaces. Our extensive experimental study on eNTERFACE database and RML database demonstrates the feasibility of the proposed multimodal emotion recognition framework based on integrated analysis of speech and facial expression. The experimental results show that the proposed methods are capable of providing improved performance. The comparison with other methods shows that the proposed two-stage fusion platform outperforms the traditional algorithms in terms of both accuracy and reliability.