Is There Correlation Between the Estimated and True Classification Errors in Small-Sample Settings?

The validity of a classifier model, consisting of a trained classifier and it estimated error, depends upon the relationship between the estimated and true errors of the classifier. Absent a good error estimation rule, the classifier-error model lacks scientific meaning. This paper demonstrates that in high-dimensionality feature selection settings in the context of small samples there can be virtually no correlation between the true and estimated errors. This conclusion has serious ramifications in the domain of high-throughput genomic classification, such as gene-expression classification, where the number of potential features (gene expressions) is usually in the tens of thousands and the number of sample points (microarrays) is often under one hundred.