Automatic word spacing using syllable n-grame and word statistics [n-grame read n-gram]

In this study, we have proposed an automatic word spacing system for the Korean language, which uses syllable n-gram and word statistics extracted from a large amount of processed corpora. The optimal spacing points of a sentence are decided mainly by using the Viterbi algorithm. As the statistical studies performance is sensitive to the training corpus and shows data sparseness problem, we have tried to enlarge the training corpora, used parameters found by examining test data and proposed an adjusting method of the ´longest match strategy´ based on the viable prefix. These increase the system´s accuracy. Our corpora, covering various language registers, were made up of 33643884 words. The pilot test was conducted with test data derived from different sources. 94.24% precision in word-unit correction were obtained on average for spacing test data.