Classifier error estimator performance in a Bayesian context

A classical approach to evaluating the accuracy of a classifier error estimator involves fixing the true distribution and averaging performance over the corresponding sampling distribution. We may evaluate marginal and mixed moments of the true and estimated errors, as well consider joint characteristics in terms of RMS or even the complete joint density. Since performance is averaged over the samples, any such analysis must be relative to a classification and error estimation rule pair. However, a new approach to evaluating error estimation accuracy has emerged from a Bayesian framework for classification, where we fix the sample itself and average over all distributions in the Bayesian model. It thus becomes possible to evaluate performance precisely for the designed classifier and obtained error estimate, resulting in a practical sample-conditioned MSE performance measure. In this article, we discuss advantages of the new Bayesian approach and fundamental differences between the classical and Bayesian methodologies.