Speechreading using shape and intensity information

We describe a speechreading system that uses both shape information from the lip contours and intensity information from the mouth area. Shape information is obtained by tracking and parameterizing the inner and outer lip boundary in an image sequence. Intensity information is extracted from a grey level model, based on principal component analysis. In comparison to other approaches, the intensity area deforms with the shape model to ensure that similar object features are represented after non-rigid deformation of the lips. We describe speaker independent recognition experiments based on these features and hidden Markov models. Preliminary results suggest that similar performance can be achieved by using either shape or intensity information and slightly higher performance by their combined use