Exact Sample Conditioned MSE Performance of the Bayesian MMSE Estimator for Classification Error—Part II: Consistency and Performance Analysis

In Part I of a two part study on the MSE performance of Bayesian error estimation, we have derived analytical expressions for MSE conditioned on the sample for Bayesian error estimators and arbitrary error estimators in two Bayesian models: discrete classification with Dirichlet priors and linear classification of Gaussian distributions with normal-inverse-Wishart priors. Here, in Part II, we examine the consistency of Bayesian error estimation and provide several simulation studies that illustrate the concept of conditional MSE and how it may be used in practice. A salient application is censored sampling, where sample points are collected one at a time until the conditional MSE reaches a stopping criterion.