Parameterized online quasi-Newton training for high-nonlinearity function approximation using multilayer neural networks

Recently, the improved online (stochastic) quasi-Newton method was developed for neural network training improving the gradient of error function. The gradient was calculated by a training sample in the online method, but the gradient of improved online one was calculated by variable training samples which were automatically increased from one to all samples as quasi-Newton iteration progressed. That is, the improved algorithm gradually changed from online to batch methods during iteration. The algorithm was efficient, and provided high quality training solutions regardless of initial values compared with online and batch methods. This paper proposes a novel robust training algorithm based on quasi-Newton in which online and batch error functions are associated by a weighting coefficient parameter. This means that the transition from the online method to the batch one is parameterized in quasi-Newton iteration in the same concept as the above improved algorithm. Furthermore, an analogy between the proposed algorithm and Langevin one is considered. Langevin algorithm is a gradient-based continuous optimization method using Simulated Annealing concept. The proposed algorithm is employed for robust neural network training purpose. Neural network training for some benchmark problems with high-nonlinearity is presented to demonstrate the validity of proposed algorithm. The new training algorithm achieves more accurate and robust training results than the other quasi-Newton based training algorithms.