Note on generalization, regularization and architecture selection in nonlinear learning systems

The author proposes a new estimate of generalization performance for nonlinear learning systems called the generalized prediction error ( GPE) which is based upon the notion of the effective number of parameters p_eff(λ). GPE does not require the use of a test set or computationally intensive cross validation and generalizes previously proposed model selection criteria (such as GCV, FPE, AIC, and PSE) in that it is formulated to include biased, nonlinear models (such as back propagation networks) which may incorporate weight decay or other regularizers. The effective number of parameters p_eff(λ) depends upon the amount of bias and smoothness (as determined by the regularization parameter λ) in the model, but generally differs from the number of weights p. Construction of an optimal architecture thus requires not just finding the weights wˆ_λ* which minimize the training function U(λ, w) but also the λ which minimizes GPE(λ)