A hybrid approach to adapting acoustic and pronunciation models for non-native speech recognition

In this paper, we propose a hybrid model adaptation approach that combines pronunciation and acoustic model adaptation methods in order to improve the performance of nonnative automatic speech recognition (ASR). Specifically, the hybrid model adaptation can be performed in two ways; at a state-tying level or a triphone-modeling level. In both methods, we first analyze the pronunciation variant rules of non-native speech and then classify each rule as either a pronunciation variant or an acoustic variant. The state-tying level method then adapts pronunciation models by adding variant pronunciations from the non-native speech and acoustic models by tying the states of triphone acoustic models using the acoustic variants. Conversely, the triphone-modeling level method adapts pronunciation models in the same way as the state-tying level method, re-estimates the triphone acoustic models using the adapted pronunciation models, and clusters the states of triphone acoustic models using the acoustic variants. From Korean-spoken English speech-recognition experiments, it is shown that the proposed hybrid acoustic and pronunciation model adaptation approach using the state-tying level method and the triphone-modeling level method can relatively reduce the average word error rates (WERs) by 16.07% and 20.94%, respectively, when compared to a baseline ASR system.