Jensen-Shannon Divergence in Ensembles of Concurrently-Trained Neural Networks

Ensembles of neural networks have been the focus of extensive studies over the past two decades. Effectively encouraging diversity remains a key element in yielding improved performance from such ensembles. Negatively correlated learning (NCL) has emerged as a promising framework for concurrently training an ensemble of learners while emphasizing the cooperation among them. The NCL methodology relies on negatively correlating the errors of the learners as means of diversifying their outputs. In this paper, we extend this framework by employing the Jensen-Shannon divergence (JSD) - an information-theoretic measure of similarity between probability distributions - as a richer measure of diversity between learners. It is argued that for classification problems, utilizing the JSD is more appropriate than negatively correlating the errors. We analyze the new formulation and derive an upper bound on the parameter that balances accuracy and diversity among the learners. Simulation results applied to standardized benchmarks clearly demonstrate the advantages of the proposed method.