Some fundamental issues in ensemble methods

The ensemble paradigm for machine learning has been studied for more than two decades and many methods, techniques and algorithms have been developed, and increasingly used in various applications. Nevertheless, there are still some fundamental issues remaining to be addressed, and an important one is what factors affect the accuracy of an ensemble, and to what extent they do, which is thus taken as the main topic of this paper. The factors studied include the accuracy of individual models, the diversity among the individual models in an ensemble, decision-making strategy, and the number of the members used for constructing an ensemble. This paper firstly describes the conceptual and theoretical analyses on these factors, and then presents the possible relationships between them. The experiments have been conducted by using some benchmark data sets and some typical results are presented in the paper.