Color-based mouth shape tracking for synthesizing realistic facial expressions

Mouth shape analysis and synthesis play an important role for realistic facial expression generation. The conventional deformable template-based method requires a fairly accurate initial localization of the template because the energy minimization process only finds a local minimum. We present a new method for accurate mouth corner detection and mouth shape estimation using color information. The proposed method can deal with a variety of shapes for both open and closed positions of the mouth. Accurate mouth corner detection and mouth status determination (open and closed) increases the accuracy of template initialization. The extracted mouth shape parameters can be used for synthesizing realistic virtual facial expressions in model based coding. Experiments on video sequences demonstrate the advantage of the proposed algorithm.