Distinctive characteristics of a metric using deviations from Poisson for feature selection

In the previous paper (Ogura, H., Amano, H., & Kondo, M. (2009). Feature selection with a measure of deviations from Poisson in text categorization. Expert Systems with Applications, 36, 6826–6832.), we proposed a new metric, χ P 2 , for selecting features in text classification which estimates term importance based on how largely the probability distribution of a considered term deviates from the standard Poisson distribution. In this study, to establish the validity and advantage of using χ P 2 , we conducted experiments of automatic text classification on 20 NewsGroups data collection with binary setting. In the experiments, other three metrics for feature selection, i.e., Gini index, χ 2 statistic and information gain, were also used for comparison. From the results, it was confirmed that χ P 2 and Gini index are much better than χ 2 statistic and information gain in terms of F 1 performance when they handle imbalanced data set. Furthermore, through another experiment in which the degree of imbalance in class distribution was explicitly controlled, we clarified that the origin of the superiority of χ P 2 and Gini index is their ability to pick up suitable negative features in imbalanced data set. The ability of these two metrics to select suitable negative features is explained based on the analysis of their limiting behaviors at some extreme cases.