A New Entropy-based Vocabulary Optimization Approach for Chinese Language Modeling

This paper proposed a new entropy-based vocabulary optimization approach for Chinese language modeling. This approach aims to directly optimize the language model by extending the vocabulary, that is, to minimize the character perplexity of the language model. A new criterion for new words selection was developed based on the character perplexity metric. A fast computing method and a simple divide-and-conquer method were proposed to deal with very large corpus. Experiments showed about 3% character perplexity reduction and 3% character error rate reduction in a speech recognition task. Comparison experiments were also conducted to compare with other approaches.