Scaling Conditional Random Field with Application to Chinese Word Segmentation

As a powerful sequence labeling model, conditional random field (CRF) has been applied to a number of natural language processing (NLP) tasks successfully. However, the high complexity of CRF training only allows a very small tag (or label)1 set, because the training becomes intractable as the tag set enlarges. This paper proposes an improved decomposed training and joint decoding algorithm for CRF learning. Instead of training a single CRF model for all tags, it trains a binary sub-CRF independently for each tag. A predicted tag sequence is then produced by a joint decoding algorithm based on the probabilistic output of all sub-CRFs involved. To test its effectiveness, this approach is applied to tackle Chinese word segmentation (CWS) as a character tagging problem. Our evaluation shows that it can reduce time and memory cost by 20-39% and 44-50%, respectively, without any significant performance loss on various large-scale data sets.