DocumentCode :
11645
Title :
Robust Audio-Visual Speech Recognition Under Noisy Audio-Video Conditions
Author :
Stewart, Darryl ; Seymour, Rowan ; Pass, Adrian ; Ji Ming
Author_Institution :
Queen´s Univ. of Belfast, Belfast, UK
Volume :
44
Issue :
2
fYear :
2014
fDate :
Feb. 2014
Firstpage :
175
Lastpage :
184
Abstract :
This paper presents the maximum weighted stream posterior (MWSP) model as a robust and efficient stream integration method for audio-visual speech recognition in environments, where the audio or video streams may be subjected to unknown and time-varying corruption. A significant advantage of MWSP is that it does not require any specific measurements of the signal in either stream to calculate appropriate stream weights during recognition, and as such it is modality-independent. This also means that MWSP complements and can be used alongside many of the other approaches that have been proposed in the literature for this problem. For evaluation we used the large XM2VTS database for speaker-independent audio-visual speech recognition. The extensive tests include both clean and corrupted utterances with corruption added in either/both the video and audio streams using a variety of types (e.g., MPEG-4 video compression) and levels of noise. The experiments show that this approach gives excellent performance in comparison to another well-known dynamic stream weighting approach and also compared to any fixed-weighted integration approach in both clean conditions or when noise is added to either stream. Furthermore, our experiments show that the MWSP approach dynamically selects suitable integration weights on a frame-by-frame basis according to the level of noise in the streams and also according to the naturally fluctuating relative reliability of the modalities even in clean conditions. The MWSP approach is shown to maintain robust recognition performance in all tested conditions, while requiring no prior knowledge about the type or level of noise.
Keywords :
audio streaming; speech recognition; video streaming; audio stream; dynamic stream weighting approach; fixed-weighted integration approach; large XM2VTS database; maximum weighted stream posterior model; noisy audio-video conditions; speaker-independent audio-visual speech recognition; stream integration method; stream weights; time-varying corruption; video stream; Automatic speech recognition; human computer interaction; speech recognition;
fLanguage :
English
Journal_Title :
Cybernetics, IEEE Transactions on
Publisher :
ieee
ISSN :
2168-2267
Type :
jour
DOI :
10.1109/TCYB.2013.2250954
Filename :
6495474
Link To Document :
بازگشت