Ensemble Bayesian Model Averaging in Genetic Programming

This paper considers the general problem of function estimation via Genetic Programming (GP). Data analysts typically select a model from a population of models, and then proceed as if the selected model had generated the data. This approach ignores the uncertainty in model selection, leading to over-confident inferences and lack of generalisation. We adopt a coherent method for accounting for this uncertainty through a weighted averaging of all models competing in a population of GP. It is a principled statistical method for postprocessing a population of programs into an ensemble, which is based on Bayesian Model Averaging (BMA). Under two different formulations of BMA, the predictive probability density function (PDF) of a response variable is a weighted average of PDFs centered around the individual predictions of component models that take the form of either standalone programs or ensembles of programs. The weights are equal to the posterior probabilities of the models generating the predictions, and reflect the models´ skill on the training dataset. The method was applied to a number of synthetic symbolic regression problems, and results demonstrate that it generalises better than standard methods for model selection, as well as methods for ensemble construction in GP.