Simplified and gradual information control for improving generalization performance of multi-layered neural networks

The present paper aims to develop a new type of information-theoretic method in which the method is simplified as much as possible by decomposing the learning procedures and gradual information control is used for training multi-layered neural networks. The information-theoretic methods have been successfully applied to the training of neural networks with two main problems, namely, the use of strong inhibition and complicated learning procedures. The strong inhibition on neurons may degrade the performance of neural networks. In addition, complicated learning procedures make it hard to apply the methods to the large-scaled data. Thus, the present paper tries to propose a new information-theoretic method without strong inhibition and to simplify the learning procedures by decomposing them into independent components. In addition, information is gradually increased for the higher layers to transfer smoothly information from the lower to the upper layers. The method was applied to the cardiotocography data set. The experimental results showed that the method could increase collectively information content by decreasing information for each neuron. This information change was found to be related to improved generalization.