The analysis on mistaken segmentation of Tibetan words based on statistical method

Author

Congjun Long ; Yiyong Lan ; Xiaobin Zhao

Author_Institution

Nat. Languages Resource Monitoring & Res. Center of Minority Language Branch, Minzu Univ. of China, Beijing, China

fYear

2014

fDate

20-22 Oct. 2014

Firstpage

74

Lastpage

77

Abstract

In this paper, by using the Tibetan word segmentation system, IEA-TWordSeg, the authors attempt segmentation of the total 1271 sentences in the closed set and 1000 sentences in an open set. The accuracy of testing is 99.54% and 92.41% respectively. The authors describe the wrong segmentation types as well as the causes of the mistakes, and demonstrate the proportion of different types of segmentation errors. The purpose of the article is to provide clues for those who intend to improve the accuracy of Tibetan word segmentation system.

Keywords

natural language processing; statistical analysis; word processing; IEA-TWordSeg; Tibetan word segmentation system; Tibetan words; mistaken segmentation; segmentation error; statistical method; Accuracy; Educational institutions; Information processing; Labeling; Testing; Training; Writing; Tibetan word segmentation; disambiguation segmentation; the segmentation errors; variant written form;

fLanguage

English

Publisher

ieee

Conference_Titel

Asian Language Processing (IALP), 2014 International Conference on

Conference_Location

Kuching

Type

conf

DOI

10.1109/IALP.2014.6973513

Filename

6973513