Probability of error, maximum mutual information, and size minimization of neural networks

An upper bound of the Bayes error probability is used as a generalization performance criterion for supervised neural network classifiers. It is shown that the maximization of the mutual information is equivalent to the minimization of this bound, and leads to a direct implementation of the Bayes framework for classification. This criterion is used both in training neural networks and in minimizing their size by adaptive pruning. A top-down heuristic for adaptively pruning nodes (weights) is proposed. The approach is applied both on probabilistic neural networks and the multilayer perceptron. Two benchmark problem results are given, verifying the validity of the approach