On jointly learning the parameters in a character-synchronous integrated speech and language model

A joint learning algorithm is proposed in this paper to enhance the integrated speech and language model operating in the character-synchronous mode. A character-synchronous approach is first proposed to integrate speech and language information, including morphology and parts-of-speech, for ranking the candidates right after each Chinese character is uttered. Since the search space is cut down very efficiently by applying high-level knowledge in early time, the character-synchronous score function enables our system to operate in real time. To further enhance system performance, a joint learning algorithm is then derived to adjust all parameters of the speech and language processing modules simultaneously, according to their contributions to the overall discrimination power, to minimize the error rate. The proposed approaches are compared with a baseline system, which directly couples an HMM-based speech recognizer with a bigram language model. The performance of 75.71% character accuracy rate is obtained for the baseline system when it is tested on the task of recognizing 1000 Chinese spoken sentences with a very large vocabulary (90495 words) in the speaker-independent isolated-character mode. The character accuracy rate is improved to 88.26% with the character-synchronous approach, and 94.16% after the joint learning algorithm is applied