A new acceleration technique for the backpropagation algorithm

An adaptive momentum algorithm which can update the momentum coefficient automatically in every iteration step is presented. The basic idea comes from the optimal gradient method. It is very difficult to obtain the optimal gradient vector by analytical methods, but it can be proven that the optimal gradient vectors in two successive iteration steps are orthogonal. Based on this property, one can use the Gram-Schmidt orthogonalization method to ensure the orthogonality of the successive gradient vectors. The result of this process is equivalent to adding a momentum term to the standard backpropagation algorithm. The momentum coefficient is updated automatically in every iteration. Numerical simulations show that the adaptive momentum algorithm can eliminate possible divergent oscillations during the initial training, and can also accelerate the learning process and result in a lower error when the final convergence is reached