Principal hidden unit analysis: generation of simple networks by minimum entropy method

In this paper, a principal hidden unit analysis with entropy minimization is proposed to obtain a simple or fundamental structure from original complex structures. The principal hidden unit analysis is composed of four steps: 1) the entropy, defined with respect to the hidden unit activity, is minimized; 2) several principal hidden units are selected, according to R-index, representing the strength of the response of hidden units to input patterns; 3) the performance of the obtained principal network is examined with respect to the error or generalization; and 4) the internal representation of the obtained principal network must appropriately be interpreted. Applied to a symmetry problem, it was confirmed that by using entropy method, a small number of principal hidden units were selected. With these principal hidden units, principal networks were constructed. The internal representation can be interpreted especially for simple problems.