Minimum α-information strategy for the interpretation of the network behaviors and the improved generalization

Proposes a minimum α-information strategy for the explicit interpretation of the network behaviors and for the improved generalization performance. The α-information is defined by the difference between Shannon and Renyi entropy. The α-information minimization can be translated into entropy maximization and entropy minimization for hidden units in term of Shannon entropy. Thus, the α-information minimization forces hidden units to have a maximum information content or a minimum information content, depending on the importance of the hidden units. For the interpretation of the network behaviors, we have only to see a small number of maximum information hidden units, ignoring the minimum information hidden units. In addition, by minimizing the α-information, the unnecessary information can be eliminated, leading to the better generalization. The authors applied the α-information minimization to the inference of the sonority of the artificial language. Experimental results explicitly confirmed that the explicit internal representation could be obtained and the generalization could significantly be improved