An adaptive step size for backpropagation using linear lower bounding functions

An adaptive step size is presented for the backpropagation algorithm in feedforward neural nets using linear lower bounding functions. Basically, a linear lower bounding function (LLBF) for a given function over an interval is a linear function that lies below the given function and matches the original function value at one end point. To search for an adaptive step size, an LLBF for the error function, which is expressed in terms of the step size, is derived. Since the error in a neural net can never be smaller than zero, it is plausible not to take a step larger than the step size when the associated LLBF reaches zero. In the paper, an adaptive learning algorithm based on the above idea is given. Numerical examples are used to illustrate its feasibility and to compare it with some previous results