Automatic learning rate optimization by higher-order derivatives

Automatic optimization of learning rate is a central issue to improving the efficiency and applicability of backpropagation learning. In this paper techniques have been investigated, which explore the first four derivatives of the learning rate of backpropagation error surface. The derivatives are derived from an extended feedforward propagation procedure and can be calculated in an iterative manner. The near-optimal dynamic learning rate is obtained with only a moderate increase in computational complexity at each iteration, scaling like the plain backpropagation algorithm (BPA), but the proposed method achieves rapid convergence and very significant gains in running time savings to at least an order of magnitude as compared with the BPA