Lip movement synthesis from speech based on hidden Markov models

Speech intelligibility can be improved by adding lip image and facial image to speech signal. Thus the lip image synthesis plays an important role to realize a natural human-like face of computer agents. Moreover the synthesized lip movement images can compensate lack of auditory information for hearing impaired people. We propose a novel lip movement synthesis method based on mapping from input speech based on Hidden Markov Model (HMM). This paper compares the HMM-based method and a conventional method using vector quantization (VQ). In the experiment, error and time differential error between synthesized lip movement images and original ones are used for evaluation. The result shows that the error of the HMM based method is 8.7% smaller than that of the VQ-based method. Moreover, the HMM-based method reduces time differential error by 32% than the VQ´s. The result also shows that the errors are mostly caused by phoneme /h/ and /Q/. Since lip shapes of those phonemes are strongly dependent on succeeding phoneme, the context dependent synthesis on the HMM-based method is applied to reduce the error. The improved HMM-based method realizes reduction of the error (differential error) by 10.5% (11%) compared with the original HMM-based method