Comparison of low- and high-level visual features for audio-visual continuous automatic speech recognition

We compare two different groups of visual features that can be used in addition to audio to improve automatic speech recognition (ASR), high- and low-level visual features. Facial animation parameters (FAPs), supported by the MPEG-4 standard for the visual representation of speech, are used as high-level visual features. Principal component analysis (PCA) based projection weights of the intensity images of the mouth area are used as low-level visual features. PCA is also applied on the FAPs. We develop an audio-visual ASR (AV-ASR) system and compare its performance for two different visual feature groups, following two approaches. The first approach assumes the same dimensionality for both high- and low-level visual features, while, in the second approach, the percentage of statistical variance described by the visual features used is the same. Multi-stream hidden Markov models (HMMs) and a late integration approach are used to integrate audio and visual information and perform continuous AV-ASR experiments. Experiments were performed at various SNRs (0-30 dB) with additive white Gaussian noise on a relatively large vocabulary database (approximately 1000 words). Conclusions are drawn on the trade off between the dimensionality of the visual features and the amount of speechreading information contained in them and its influence on the AV-ASR performance.