Dynamic sample size selection based quasi-Newton training for highly nonlinear function approximation using multilayer neural networks

This paper describes a novel robust training algorithm based on quasi-Newton iteration. The size of training samples for each iteration is dynamically and analytically determined by variance estimates during the computation of its gradient in the mini-batch based online training methodology. Furthermore, the size of mini-batch is controlled by a parameter to ensure that the number of samples in a mini-batch changes from a portion of samples (online) to all ones (batch) as quasi-Newton iteration progressed. As a result, the iteration during online mode can be shortened compared with previous quasi-Newton-based methods in which the gradient of error function for the training step was improved.