Lipreading from color motion video

We have designed and implemented a lipreading system which recognises isolated words using only color motion video of human lips (without acoustic data). The lipreading system performs color motion video recognition using “snakes” (active contour models), principal component analysis (PCA), and hidden Markov models (HMM). The snake algorithm and PCA are used to extract two sets of visual features from every frame (image) in the video sequence. The snake algorithm looks for contour features in the geometric space, while PCA seeks principal components in the eigenspace. An HMM recognizer is used to train and recognise a sequence of the combined visual features. With the visual information alone, we were able to achieve 94% recognition accuracy for 10 isolated words of a single speaker without using any special marker or lipstick