A comparison between criterion functions for linear classifiers, with an application to neural nets

The error rates of linear classifiers that utilize various criterion functions are investigated for the case of two normal distributions with different variances and a priori probabilities. It is found that the classifier based on the least mean squares (LMS) criterion often performs considerably worse than the Bayes rate. The perceptron criterion (with suitable safety margin) and the linearized sigmoid generally lead to lower error rates than the LMS criterion, with the sigmoid usually the better of the two. Also investigated are the exceptions to the general trends: only if one class is known to have much larger a priori probability or variance than the other should one expect the LMS or perceptron criteria to be slightly preferable as far as error rate is concerned. The analysis is related to the performance of the back-propagation (BP) classifier, giving some understanding of the success of BP. A neural-net classifier, the adaptive-clustering classifier, suggested by this analysis is compared with BP (modified by using a conjugate-gradient optimization technique) for two problems. It is found that BP usually takes significantly longer to train than the adaptive-clustering technique