Improving Chinese Handwriting Recognition by Fusing Speech Recognition

The texts recognized from a piece of Chinese handwriting and the speech corresponding to the same handwriting, respectively, are complementary to each other. A better text, thus, can be obtained by fusing the two texts, since a fused text can contain more semantic information. The fused text should cover all characters in the two texts and these characters can be arranged in any order. There are four ways to select or pass over a character in the two texts. This paper proposes to formulate how to fuse them properly as an optimization problem, and solves it using a dynamic programming algorithm. The solution space for the fused text is represented as a directed graph with levels, which number is equal to the sum of character numbers of the two texts. The optimal fused texts correspond to the optimal paths in the graph. Experimental results demonstrate the proposed approach is effective and robust.