Bagging very weak learners with lazy local learning

Bagging predictors have shown to be effective especially when the learners used to train the base classifiers are weak. In this paper, we argue that for very weak (VW) learners, such as DecisionStump, OneR, and SuperPipes, the base classifiers built from boostrap bags are strongly correlated with each other. As a result, a simple bagging (SB) predictor built on such VW learners has very little improvement compared to a single classifier trained from the same data. Alternatively, we propose a Local Lazy Learning based bagging approach (L³B), where base learners are trained from a small instance subset surrounding each test instance. More specifically, given a test instance x, L³B first discovers x¿s k nearest neighbours, and then applies progressive sampling to the selected neighbours to train a set of base classifiers, by using a given VW learner. At the last stage, x is labeled as the most frequently voted class of all base classifiers. Experimental results on 32 real-world datasets, including two high dimensional gene expression datasets, demonstrate that L³B significantly outperforms SB for building accurate classifier ensemble models for VW learners.