A validation method for comparing classifiers on imbalanced datasets

In this study, to compare the robustness and learning capability of the classifiers on imbalanced datasets, a cross validation method that generates class-imbalanced training sets is proposed. The method will also be used to evaluate the accuracies of methods developed for dealing with the class-imbalance problem. The proposed method is used to generate imbalanced datasets from three biomedical datasets. Then, k-Nearest Neighbor, Support Vector Machines and Multi Layer Perceptron classifiers are compared using various settings of their hyper-parameters that affect their complexities. The experimental results show that SVMs are simply the most robust of all when applied to imbalanced datasets.