Speech-to-video synthesis using facial animation parameters

The presence of visual information in addition to audio could improve speech understanding in noisy environments. This additional information could be especially useful for people with impaired hearing who are able to speechread. This paper focuses on the problem of synthesizing the facial animation parameters (FAPs), supported by the MPEG-4 standard for the visual representation of speech, from a narrowband acoustic speech (telephone) signal. A correlation hidden Markov model (CHMM) system for performing visual speech synthesis is proposed. The CHMM system integrates an independently trained acoustic HMM (AHMM) system and a visual HMM (VHMM) system, in order to realize speech-to-video synthesis. Analyzing the synthesized FAPs and computing the time alignment errors perform objective experiments. Time alignment errors are reduced by 40.5% compared to the conventional temporal scaling method.