New Word Extraction Utilizing Google News Corpuses for Supporting Lexicon-based Chinese Word Segmentation Systems

This study proposes a novel statistics-based scheme for new word extraction based on Google news to promote the word identification ability for the lexicon-based Chinese word segmentation systems. To extract news words from the corpuses of news and incrementally add them into the lexicon for the lexicon-based Chinese word segmentation systems provides benefits in terms of automatically constructing a professional lexicon of news and enhancing word identification ability. Compared with another proposed method, the experimental results indicated that the proposed new word extraction scheme not only can more correctly retrieve news words from the categorized corpuses of Google news, but also obtain has larger amount of new words.