A comparison of weight elimination methods for reducing complexity in neural networks

Three methods are examined for reducing complexity in potentially oversized networks. These consists of either removing redundant elements based on some measure of saliency, adding a further term to the cost function penalizing complexity, or observing the error on a further, validation set of examples, and then stopping training as soon as this performance begins to deteriorate. It was demonstrated on a series of simulation examples that all of these methods can significantly improve generalization, but their performance can prove to be domain dependent