Methodology and operation for extraction of Chinese segmentation units

Chinese is written with the character of no space or other word delimiters. Chinese word segmentation (CWS) is the first step for Chinese language processing. Generally, words and fixed phrases(idioms, named entity) can be tagged successfully. However, besides words and fixed phrases, segmentation units should be tagged too. In the segmentation specification written by Liu Yuan, segmentation units are defined as “integrated closely and used steadily” phrases. There is no explicit definition to judge whether a phrase is integrated closely and used steadily or not, and no operable method to extract segmentation units till now. This paper puts forwards the principle to evaluate whether a phrase meets the standard of close integration and steady usage first, and then brings forward a detailed method to extract segmentation unit. We use a hybrid method to do the extraction task, which includs calculation of frequency, mutual information, entropy, and linguistic information of both syntactic rules and semantic explaination. We extract segmentation units in the frame of multi-features and do manual evaluation. Experiment shows that the approach is effective. Our job can provide a desirable list of segmentation units for NLP implication.