An Exemplar-Based Hidden Markov Model with Discriminative Visual Features for Lipreading

In this paper, we address an exemplar-based hidden Markov model (HMM) that represents the lip motion activity using visual cues for lipreading. The discriminative visual features including the geometric shape parameters and contour-constrained spatial histogram are selected for representing each lip frame. Then, a set of exemplars associated with the HMM is learned jointly to serve as a typical representation of a speech utterance. Based on these exemplars, the high-dimensional frame features are transformed to the lower dimensional ones, namely Frame to Exemplar Distance (FED) vector. Subsequently, a continuous HMM is trained using such FED vector sequences for learning and recognition. Experiments show the promising results.