A new Neural Network pruning method based on the singular value decomposition and the weight initialisation

In this paper, we present an efficient procedure to determine the optimal hidden unit number of a feed-forward multi-layer Neural Network (NN) using the singular value decomposition (SVD) taking into account the function to be approximated by the NN and the initial values of the updating weights. The SVD is used to identify and eliminate redundant hidden nodes. Minimizing redundancy gives smaller networks, producing models that generalize better and thus eliminate the need of using cross-validation to avoid overfitting. Using this procedure we obtain a final model with fewer adjustable parameters and more accurate predictions than a network model with a fixed, a priori determined, size. We show these performances by applying this procedure to several problems such as function approximation and image recognition.