Title :
Exploiting a ‘gaze-Lombard effect’ to improve ASR performance in acoustically noisy settings
Author :
Cooke, Neil ; Ao Shen ; Russell, Matthew
Author_Institution :
Sch. of Electron. & Electr. & Comput. Eng., Univ. of Birmingham, Birmingham, UK
Abstract :
Previous use of gaze (eye movement) to improve ASR performance involves shifting language model probability mass towards the subset of the vocabulary whose words are related to a person´s visual attention. Motivated to improve Automatic Speech Recognition (ASR) performance in acoustically noisy settings by using information from gaze selectively, we propose a `Selective Gaze-contingent ASR´ (SGC-ASR). In modelling the relationship between gaze and speech conditioned on noise level - a `gaze-Lombard effect´-simultaneous dynamic adaptation of acoustic models and the language model is achieved. Evaluation on a matched set of gaze and speech data recorded under a varying speech babble noise condition yields WER performance improvements. The work highlights the use of gaze information in dynamic model-based adaptation methods for noise robust ASR.
Keywords :
acoustic noise; gaze tracking; probability; speech recognition; ASR performance; SGC-ASR; WER performance improvements; acoustic models; acoustically noisy settings; automatic speech recognition performance; dynamic model-based adaptation methods; eye movement; gaze information; gaze-Lombard effect; language model probability mass; noise level; selective gaze-contingent ASR; speech babble noise condition; speech data; visual attention; vocabulary; Acoustic noise; Adaptation models; Noise; Noise measurement; Speech; Visualization; ASR; Acoustic Model adaptation; Language Model adaptation; Mutual information; acoustic noise; gaze; noise robust ASR. eye movement; speech; visual attention;
Conference_Titel :
Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on
Conference_Location :
Florence
DOI :
10.1109/ICASSP.2014.6853899
