Fast backpropagation learning using steep activation functions and automatic weight reinitialization

Several backpropagation (BP) learning speed-up algorithms that employ the gain parameter, i.e., steepness of the activation function, are examined to determine the effect of increased gain on learning time. It was shown by simulation that although these algorithms can converge faster than the standard BP learning algorithm on some problems, they can be unstable in convergence, i.e., they frequently fail to converge within a finite time. One main reason for this divergence is inappropriate setting of initial weights in the network. To overcome this instability, an automatic random reinitialization of the weights is proposed when convergence speed becomes very slow. BP learning algorithms with this weight reinitialization and larger initial gain (around 2 or 3) were found to be much faster and more stable in convergence than those without weight reinitialization