Automatic inference of mental states from spontaneous facial expressions

Human face is a display of mental states that reflect the true feelings of a person. In this paper, we propose a framework for the video analysis of spontaneous facial expressions using an automatic facial emotion recognition system. Regional Hidden Markov Models (RHMMs) are created to describe the states of facial attributes for eyebrows, eyes, and mouth regions registered in a video sequence. The performance results reported in the paper show that the proposed technique outperforms the designated HMM for each emotion type [1, 2] tested with the Cohn-Kanade database for the person-independent case. More importantly, we used the proposed system to infer the mental states of a person based on spontaneous facial expressions. Merit of the proposed system is validated with human based evaluations.