Title :
A Practical Approach to Resolving Combination Ambiguity in Chinese Word Segmentation
Author :
Qin, Ying ; Zhang, Suxiang ; Wang, Xiaojie
Author_Institution :
Sch. of Inf. Eng., Beijing Univ. of Posts & Telecommun.
Abstract :
In Chinese word segmentation task, combination ambiguity is one of challenges not being well settled. The main obstacle exists in the detection of ambiguous words in given texts and their proper segmentations. This paper puts forward a practical approach to automatically collecting ambiguous words and disambiguating based on maximum entropy principle. The experimental result reveals the approach of automatic collection ambiguous words can detect combination ambiguity effectively avoiding arduous manual work. As to the disambiguation based on maximum entropy, we investigate new features grounded on prior and contextual knowledge and achieve promising result
Keywords :
maximum entropy methods; natural language processing; Chinese word segmentation; combination ambiguity; contextual knowledge; maximum entropy principle; Dictionaries; Entropy; Humans; Power engineering and energy; Power engineering computing; Statistics; Telecommunication computing; Testing; Text processing;
Conference_Titel :
Signal Processing, 2006 8th International Conference on
Conference_Location :
Beijing
Print_ISBN :
0-7803-9736-3
Electronic_ISBN :
0-7803-9736-3
DOI :
10.1109/ICOSP.2006.345823