A Double Pruning Scheme for Boosting Ensembles

Ensemble learning consists of generating a collection of classifiers whose predictions are then combined to yield a single unified decision. Ensembles of complementary classifiers provide accurate and robust predictions, which are often better than the predictions of the individual classifiers in the ensemble. Nevertheless, ensembles also have some drawbacks: typically, all classifiers are queried to compute the final ensemble prediction. Therefore, all the classifiers need to be accessible to address potential queries. This entails larger storage requirements and slower predictions than a single classifier. Ensemble pruning techniques are useful to alleviate these drawbacks. Static pruning techniques reduce the ensemble size by selecting a sub-ensemble of classifiers from the original ensemble. In dynamic pruning, the querying process is halted when the partial ensemble prediction is sufficient to reach a stable final decision with a reasonable amount of confidence. In this paper, we present the results of a comprehensive analysis of static and dynamic pruning techniques applied to Adaboost ensembles. These ensemble pruning techniques are evaluated on a wide range of classification problems. From this analysis, one concludes that the combination of static and dynamic pruning techniques provides a notable reduction in the memory requirements and an improvement in the classification time without a significant loss of prediction accuracy.