A Robust Multi-Modal Emotion Recognition Framework for Intelligent Tutoring Systems

This paper presents a multi-modal emotion recognition framework that is capable of estimating the human emotional state through analyzing and fusing a number of non-invasive external cues. The proposed framework consists of a set of data analysis, feature extraction and emotion recognition modules for processing heterogeneous sensory data (e.g., visual appearance and speech) and a novel probabilistic information fusion model to accurately estimate the human emotional state. Experimental results demonstrate that the proposed emotion recognition framework can automatically and robustly recognize human emotional states. Our results also proof that by fusing complementary information such as facial expression analysis and voice intonation analysis results, the emotion recognition performance can be boosted and outperform each individual modal analysis. The proposed emotion recognition framework can be integrated into existing Intelligent Tutoring Systems (ITSs) for improving the effectiveness of the learning systems by providing feedbacks to the ITSs.