A learning algorithm for multilayered neural networks: a Newton method using automatic differentiation

Summary form only given, as follows. A learning algorithm for multilayered neural networks which is implemented by a Newton method using automatic differentiation was compared to the back-propagation method. It has been thought that the computational cost for obtaining second-order derivatives of an error function is very high, and that a system of linear equations (the Newton equations) cannot be solved practically for large-scale neural networks. However, a forward method of automatic differentiation enables one to calculate the product of the Hessian of the error function and a search direction vector, without calculation of the Hessian itself, with a cost proportional to the cost for the error function. Therefore, even if the network is large, the Newton equations can be solved. Computer simulations show that this method converges to the solutions more rapidly than the back-propagation method