A complexity based Silent Pruning Algorithm

This paper presents a Lempel Ziv Complexity (LZC) based pruning algorithm, called Silent Pruning Algorithm (SPA), for designing artificial neural networks (ANNs). This algorithm prunes hidden neurons during the training process of ANNs according to their ranks computed with LZC. LZC extracts the number of unique patterns in a time sequence as a measure of rank. As a result, it is expected that LZC computes idleness or activeness of a hidden unit in training. The pruning is done on hidden nodes of three layered feed-forward neural network based on lowest LZC (lowest rank) of hidden neurons. This algorithm is similar to yet different from other so far existing pruning algorithms. One of the sub goals of this method is to quantify the complexity and topology of ANN. The SPA not only prunes hidden units but also facilitates to maintain complexity. The proposed SPA has been tested on a number of challenging benchmark problems in machine learning and ANNs, including Cancer, Diabetes, Heart disease, Card, and Glass identification problems. In order to justify the effectiveness of SPA we have developed a method which prunes hidden units randomly. In addition SPA is compared with variance analysis of sensitivity information based pruning method. The experimental results show that SPA can design compact ANN architectures with good generalization ability. The results also show that facilitating complexity and suppressing simplicity help the training to produce resourceful and functional network.