Title :
The analysis on mistaken segmentation of Tibetan words based on statistical method
Author :
Congjun Long ; Yiyong Lan ; Xiaobin Zhao
Author_Institution :
Nat. Languages Resource Monitoring & Res. Center of Minority Language Branch, Minzu Univ. of China, Beijing, China
Abstract :
In this paper, by using the Tibetan word segmentation system, IEA-TWordSeg, the authors attempt segmentation of the total 1271 sentences in the closed set and 1000 sentences in an open set. The accuracy of testing is 99.54% and 92.41% respectively. The authors describe the wrong segmentation types as well as the causes of the mistakes, and demonstrate the proportion of different types of segmentation errors. The purpose of the article is to provide clues for those who intend to improve the accuracy of Tibetan word segmentation system.
Keywords :
natural language processing; statistical analysis; word processing; IEA-TWordSeg; Tibetan word segmentation system; Tibetan words; mistaken segmentation; segmentation error; statistical method; Accuracy; Educational institutions; Information processing; Labeling; Testing; Training; Writing; Tibetan word segmentation; disambiguation segmentation; the segmentation errors; variant written form;
Conference_Titel :
Asian Language Processing (IALP), 2014 International Conference on
Conference_Location :
Kuching
DOI :
10.1109/IALP.2014.6973513
