Audio-lingual and Visual-facial Emotion Recognition: Towards a Bi-modal Interaction System

Towards building a multimodal affect recognition system, we have built a facial expression recognition system and a audio-lingual affect recognition system. In this paper, we present and discuss the development and evaluation process of the two subsystems, concerning the recognition of emotions from audio-lingual and visual-facial modalities. Many researchers agree that these modalities are complementary to each other and that the combination of the two can improve the accuracy in affective user models. Therefore in this paper we present a combination of two modes using multi-criteria decision making theories. The resulted system takes advantage of the strengths of each mode and is more accurate in emotion recognition.