Adaptive Iterative Learning for Classification based on Feature Selection and Combination Voting

Feature selection is an active research area in machine learning for high dimensional dataset analysis. The idea is to perform the learning process solely on the top ranked feature spaces instead of the entire original feature space, and therefore to improve the understanding of the inherent characteristics of such dataset as well as reduce the computational cost. While most of the research efforts are focused on how to select the proper features for machine learning, we studied the following important problem in this paper: can the "unimportant features" (low rank features) also provide useful information to improve the overall learning capability? In this paper, we proposed an adaptive iterative learning mechanism based on feature selection and combination voting (AdaFSCV) to address this issue. Unlike the conventional way of discarding the unselected low rank features, we iteratively build classifiers in those feature spaces as well. Such iterative process will adaptively learn information in different feature spaces, and automatically stop when one classify can not provide better information than a random guess. Finally, a probability voting algorithm is proposed to combine all the votes from different classifiers to provide the final prediction results. Simulation results on the MNIST database of handwritten digits show this method can improve the classification accuracy and robustness with certain levels of trade-off of the computational cost.