A note on the estimation of the generalization error and the prevention of overfitting [machine learning]

Valid generalization and overfitting are closely related issues in the theory of machine learning. In the context of multilayer perceptrons (MLPs) it is usually assumed that improved generalization can be achieved by reducing the functionality of the network. Overfitting is battled by cross-validation methods. A different look on these subjects is proposed by this paper. A Radial-Basis-Function (RBF) network is developed that provides statistically consistent estimates of the expected generalization error induced by arbitrary MLPs. Using the RBF network as a teacher for the MLP, a new batch backpropagation-type learning algorithm is developed that minimizes directly the estimated generalization error. The on-line version of this algorithm calls for a random generalization of the training set. It is concluded that the problem of improved generalization performance is equivalent to the problem of valid generalization of the training data set. Therefore, both problems fall within the statistical context of nonparametric density estimation