Nonlinear Hebbian rule: a statistical interpretation

Recently, the extension of Hebbian learning to nonlinear units has received increased attention. Some successful applications of this learning rule have been reported as well; however, a fundamental understanding of the capability of this learning rule is still lacking. In this paper, we pursue a better understanding of what the network is actually doing by exploring the statistical characteristics of the criterion function and interpreting the nonlinear unit as a probability integral transformation. To improve the capability of the nonlinear units, data preprocessing is suggested. A better data preprocessing leads to the development of a two-layer network which consists of linear units in the first layer and nonlinear units in the second layer. The linear units capture and filter the linear aspect of the data and the nonlinear units discover the nonlinear effects, such as clustering and other general nonlinear associations among the variables. Several potential applications are demonstrated through the simulation results given throughout this paper