Performance criteria and discrimination of extreme undersmoothing in nonparametric regression

The prediction error (average squared error) is the most commonly used performance criterion for the assessment of nonparametric regression estimators. However, there has been little investigation of the properties of the criterion itself. This paper shows that in certain situations the prediction error can be very misleading because it fails to discriminate an extreme undersmoothed estimate from a good estimate. For spline smoothing, we show, using asymptotic analysis and simulations, that there is poor discrimination of extreme undersmoothing in the following situations: small sample size or small error variance or a function with high curvature. To overcome this problem, we propose using the Sobolev error criterion. For spline smoothing, it is shown asymptotically and by simulations that the Sobolev error is significantly better than the prediction error in discriminating extreme undersmoothing. Similar results hold for other nonparametric regression estimators and for multivariate smoothing. For thin-plate smoothing splines, the prediction error׳s poor discrimination of extreme undersmoothing becomes significantly worse with increasing dimension.