Visualize speech: a continuous speech recognition system for facial animation using acoustic visemes

This paper presents an acoustic viseme based continuous speech recognition system for speech driven talking face animation. The system is developed using viseme HMMs with acoustic speech as input only. Triseme HMMs are adopted to reflect the mouth shape contexts. Visual decision trees are introduced to get robust parameter training for triseme HMMs with the limited training data. In the tree building process, methods based on lip rounding and similarity of viseme shapes are introduced to design visual questions. The results from objective and subjective evaluations show that the talking face animation based on the speech recognition system provided by this paper outperforms the conventional phoneme based one, and it is possible to obtain visually relevant speech segmentation information from acoustic speech signal only.