Online Learning Dynamics of Radial Basis Function Neural Networks near the Singularity

It has been found that strange behaviours will happen because of the singularity in the parameter space (or neuro-manifold) of hierarchical models such as feed-forward neural networks, and the learning dynamics of multilayer perceptrons near the singularity has been well discussed. In this paper, the online learning dynamics near the singularity is investigated for radial basis function (RBF) neural networks with all its unit centers, widths and output weights being continuously modified using standard gradient descent algorithm. Results show that in the case of the teacher is on the singularity, if we initiate the learning process near the singularity, then the final parameter values of hidden units are dependant on their initial values: if two hidden units are initialized with similar unit centres and widths, they will overlap; otherwise, one of the hidden units will eliminate.