An optimal line search algorithm for the conjugate gradient method

A new line search technique for the conjugate gradient (CG) method, critical point approximation (CPA), is introduced. The CPA is an elaborately revised version of the parabolic interpolation (PI). The new algorithm evaluates the function and gradient at just one point for each line search, while the conventional PI requires two points. Although the CG usually evaluates the gradient at each critical point found in the iteration, the CG with CPA approximates the gradient, using information obtained for the line search. The new algorithm is implemented in an artificial neural network program for experiment and comparison. The results show that the CG method with the new algorithm converges significantly faster than that with the conventional PI. Under an optimistic assumption, the author explains that the CPA is an optimal line search algorithm for the CG method. Issues regarding the precision and computation time of the line search are discussed.