Minimum phone error based stream weight training for mandarin audio-visual Speech recognition

Stream weight training is one of the key issues in the bimodal integration for the audio-visual speech recognition. In this paper, the audio- and video-only HMM classifiers are combined to recognize audio-visual speech recognition. More specifically, a discriminative training method is provided, in which the state-dependent stream weights are trained based on lattice rescoring by the minimum phone error using the extended Baum Welch algorithm. The proposed method is evaluated on our Mandarin large vocabulary audio-visual database. Experimental results show the proposed method has achieved significant error reduction than traditional global stream weight based approach and outperforms the minimum classification error based discriminative stream weight training method.