On evaluating performance of classifiers for rare classes

Predicting rare classes effectively is an important problem. The definition of effective classifier, embodied in the classifier evaluation metric, is however very subjective, dependent on the application domain. In this paper a wide variety of point-metrics are put into a common analytical context defined by the recall and precision of the target rare class. This enables us to compare various metrics in an objective, domain-independent manner. We judge their suitability for the rare class problems along the dimensions of learning difficulty and levels of rarity. This yields many valuable insights. In order to address the goal of achieving better recall and precision, we also propose a way of comparing classifiers directly based on the relationships between recall and precision values. It resorts to a composite point-metric only when recall-precision based comparisons yield conflicting results.