A novel approach for training small-sized multilayer perceptrons

The author present a novel approach tailored for training small-sized multilayer perceptrons. The approach presented updates a single randomly chosen weight with a stable second-order update rule, at each epoch. He discusses the analytical proof of stability and convergence at local minima. The paper identifies the situations when the approach is better than backpropagation for learning. Based on the experiments conducted, he shows four important results. First, the approach outperforms backpropagation when the network size is very small. Secondly, with slightly larger sized networks, though backpropagation tends to beat the approach when the perceptron activation function slope is low, the approach reaches better minima when the activation function slope is high. Thirdly, the approach normally reduces error when all the initial weights are equal, which is when back-propagation performs poorly. Finally, the approach tends to avoid premature saturation, during the early stages of learning, due to poorly initialized weights.