Reconsideration to pruning and regularization for complexity optimization in neural networks

The ultimate purpose of neural network design is to find an optimal network that can give good generalization performance with compact structure. To achieve this, it is necessary to control complexities of networks so as to avoid its overfitting to noisy learning data. The most popular methods for complexity control are the pruning method and the regularization method. Even though there have been many variations in the methods, the peculiar properties of each method compared to others has not been so clear. We reconsider the pruning strategy from a geometrical and statistical viewpoint, and show that the natural pruning method is in accordance with the geometrical and statistical intuition in choosing connections to be pruned. In addition, we also suggest that the regularization method should be used in combination with natural pruning in order to improve the optimization performance. We also show some experimental results supporting our suggestions.