High quality facial expression recognition in video streams using shape related information only

Person independent and pose invariant facial emotion classification is important for situation analysis and for automated video annotation. Shape and its changes are advantageous for these purposes. We estimated the potentials of shape measurements from the raw 2D shape data of the CK+ database. We used a simple Procrustes transformation and applied the multi-class SVM leave-one-out method. We found close to 100% classification performance demonstrating the relevance of details in shape space. Precise, pose invariant 3D shape information can be computed by means of constrained local models (CLM). We used this method: we fitted 3D CLM to CK+ data and derived the frontal views of the 2D shapes. Performance reached and sometimes surpassed state-of-the-art results. In another experiment, we studied pose invariance: we rendered 3D emotional database with different poses using BU 4DFE database, fitted 3D CLM, transformed the 3D shape to frontal pose and evaluated the outputs of our classifier. Results show that the high quality classification is robust against pose variations. The superior performance suggests that shape, which is typically neglected or used only as side information in facial expression categorization, could make a good benchmark for future studies.