Levenberg-Marquardt algorithm with adaptive momentum for the efficient training of feedforward networks

We present a highly efficient second order algorithm for the training of feedforward neural networks. The algorithm is based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Its implementation requires minimal additional computations compared to a standard LM iteration which are compensated, however, from its excellent convergence properties. Simulations of large scale classical neural network benchmarks are presented which reveal the power of the method to obtain solutions in difficult problems whereas other standard second order techniques (including LM) fail to converge