Pruning neural networks during training by backpropagation

For any neural network design, the network size is often chosen arbitrarily. Too large a network will tend to memorise the training patterns and thus have a poor generalisation ability. A smaller network is more efficient in computations and learning. However, too small a network may never solve the problem. In general, one would rather overestimate the network size than underestimate it. Pruning, or net pruning, is the reduction of the network size. Karnin (1990) proposed a simple procedure for pruning back-propagation trained neural networks. This paper extends the work by Karnin further and proposes a simple method of pruning during training by backpropagation