A segmentation method for crossing ambiguity string based on mutual information and t-test difference

One nodus existing in Chinese word segmentation is the ambiguity problem of which more than 85% are crossing ambiguity, therefore it is significant to decrease the error in dealing with the crossing ambiguity. Taking the advantage of the characteristics of the crossing ambiguity string, a novel method based on the mutual information and t-test difference is proposed to deal with the ambiguities in Chinese word segmentation. The mutual information which denotes the close degree of the two Chinese characters is calculated firstly. Then the t-test difference which reflected the context information of the two Chinese characters is calculated. And finally the segmentation position is obtained by using the mutual information and t-test difference. The experiment results demonstrate that the accuracy of segmentation can be effectively improved.