On the structure of the Hessian matrix in feedforward networks and second derivative methods

Adaptation in feedforward networks based on backpropagation learning is one of the most important techniques in the area of artificial neural networks. Considering properties of backpropagation learning it is possible to construct efficient adaptive first derivative algorithms. A possibility to improve this adaptation is given by using second derivatives of the error function. But a lot of problems arise when applying such algorithms. How can optimized adaptive methods with second derivatives be applied? This paper deals with investigation into the Hessian matrix in feedforward networks and its properties. Furthermore a formulation of a separated online learning algorithm using second derivatives is presented