Sparse algorithm for feed-forward neural networks

In this paper self-adjustment pruning algorithm is presented to make arbitrary feed-forward neural networks sparse. To adapt to the complex and flexible architecture of arbitrary feed-forward neural network, partial order and topology order are introduced. Based on the exemplar vector, discrete degree and similarity degree are presented. In order to search redundant neurons and connections, the criterions of redundant neurons and connections are brought up. In order to keep up the original input-output behavior it is required to modify the remaining weights and threshold locally, and create new connections sometimes. The change to the related weights and thresholds can be obtained by constructing and solving a system of linear equations. Experimental results show this algorithm is very effective and feasible. The proposed algorithm is not only faster than general sparse algorithm but also can keep up the original performance and improve the generalization. So it is not required to retrain the neural network after pruning