A Study of chi^2-test for Text Categorization

Author

Chen, Yao-Tsung ; Chen, Meng Chang

Author_Institution

Dept. of Comput. Sci. & Inf. Eng., Nat. Penghu Univ.

fYear

2006

fDate

18-22 Dec. 2006

Firstpage

305

Lastpage

308

Abstract

In this paper, we propose the chi²-classifier employing the chi²-test to test the homogeneity of two random samples of term vectors for text categorization decision. First, the properties of chi²-test for text categorization are studied. One of the advantages of chi²-test is that its significance level a is the same as the miss rate that provides a foundation for theoretical performance guarantee. The chi²-classifier also considers term aggregation and selection methods to improve the categorization performance. Generally cosine similarity with TF*IDF weighting function performs reasonably well in text categorization. However, the performance of cosine similarity depends on the given threshold value, and its categorization performance may fluctuate even near the optimal threshold value. To alleviate the problems, the chi²-classifier proposes a combination of chi²-test and cosine similarity. Extensive experiment results have verified the properties of chi²-test and performance of the combined classifier

Keywords

classification; data mining; learning (artificial intelligence); sampling methods; statistical testing; text analysis; chi²-classifier; chi²-test; cosine similarity; random sampling; term aggregation method; term selection method; text categorization; Computer science; Frequency; Information science; Machine learning; Statistical analysis; Statistical distributions; Statistics; Testing; Text categorization; Text mining; G.3g Nonparametric statistics; H.2.8.I Text mining; I.2.6.g Machine Learning;

fLanguage

English

Publisher

ieee

Conference_Titel

Web Intelligence, 2006. WI 2006. IEEE/WIC/ACM International Conference on

Conference_Location

Hong Kong

Print_ISBN

0-7695-2747-7

Type

conf

DOI

10.1109/WI.2006.27

Filename

4061382