KETW Key Terms Extraction and Term Weighting for Newsgroup Message Classification

Messages in newsgroups are hierarchically organized into several thousands of groups such as soc.religion, alt.atheism, etc. Automatic text classification can help users determine the most relevant group to post a message. However, because of the huge size of newsgroup data sets, high time complexity and low classification accuracy are great challenges for automatic message classification. To solve these problems, we propose a system named key terms extraction and term weighting (KETW) to automatically classify new messages for large newsgroup data sets. The system consists of two parts: (1) utilize the special properties of newsgroup data to extract representative terms for each group and use these terms for training rather than the whole vocabulary; (2) give terms different weights according to their importance. Our experimental results demonstrate that the technique of refining the training set reduces to 10% of the storage and still achieve good performance.