Simple addition to back-propagation learning for dynamic weight pruning, sparse network extraction and faster learning

The enhancement to the backpropagation algorithm presented results from the need to extract sparsely connected networks from networks employing product terms. The enhancement works in conjunction with the backpropagation weight update process, so that the actions of weight zeroing and weight stimulation enhance each other. It is shown that the error measure can also be interpreted as rate of weight change (as opposed to ΔW_ij), and consequently is used to determine when weights have reached a stable state. Weights judged to be stable are then compared to a zero weight threshold. Should they fall below this threshold, the weight in question is zeroed. Simulation of such a system is shown to return improved learning rates and reduce network connection requirements with respect to the optimal network solution, trained using the normal backpropagation algorithm for multi-layer perceptron (MLP), higher order neural network (HONN) and sigma-pi networks