Empirical risk minimization versus maximum-likelihood estimation: A case study

Considers a simple two class pattern classification problem from two points of view, namely that of empirical risk minimization and that of maximum-likelihood estimation. The main focus is on an exact solution for the generalization error resulting from the above two approaches, emphasizing mainly the finite sample behavior, which is very different for the two methods. Focusing on the case of normal input distributions and linear threshold classifiers, the author uses statistical mechanics techniques to calculate the empirical and expected (or generalization) errors for the maximum-likelihood and minimal empirical error estimation methods, as well as several other algorithms. In the case of spherically symmetric distributions within each class the author finds that the simple Hebb rule, corresponding to maximum-likelihood parameter estimation, outperforms the other more complex algorithms, based on error minimization. Moreover, the author shows that in the regime where the overlap between the classes is large, algorithms with low empirical error do worse in terms of generalization, a phenomenon known as over-training