Study on Good-Turing and a novel smoothing method based on real corpora for language models

In this paper, we study the well-known Good-Turing smoothing technique, which has inherited issue and propose a novel method. The smoothing method is used to resolve the zero count problems in traditional language models. Basically, there are two processes for smoothing techniques: discounting and redistributing. The statistical behavior of smoothing method is analyzed on various training data size. Several features for Good-Turing are used to improve the method. We propose novel smoothing methods based on real training data sets, which will reflect the physical behavior of smoothing method. With respect to the heuristic probability assignment used by most smoothing methods, our method is based on real training corpora. The curves of number for events with different counts can be plotted. The behavior of our method is analyzed. The empirical results are presented to compare the effectiveness of smoothing methods.