Network expansion and network compression-further discussion on structure variation methods

There has been an increasing tendency of varying the structure of networks during training processes. The author has divided the structure variation methods in feedforward neural networks into three parts-network expansion, network construction, and network compression and presents some methods. Since there are numerous papers concerning network construction, in this paper, the author only further addresses two parts: network expansion and network compression. The author groups network expansion methods into two classes: retraining all the weights and only training the added weights after the network is expanded. For network compression, the author first divides the compression process into two stages: in the first stage the author selects which connection or neuron should be moved. In this stage, the compression strategies are classified into three groups. In the second stage, the author prunes the selected one. In this stage, the author proceeds by another strategy in which the pruning methods are classified into two groups: pruning without information crosswise propagation (CP) and pruning with information CP. Combination of network expansion and network compression is generally used in practice. As a special case of combination of network expansion and network compression, a duplication problem is addressed