Constrained gradient descent and line search for solving optimization problem with elliptic constraints

Finding global minima and maxima of constrained optimization problems is an important task in engineering applications and scientific computation. In this paper, the necessary conditions of optimality will be solved sequentially using a combination of gradient descent and exact or approximate line search. The optimality conditions are enforced at each step while optimizing along the direction of the gradient of the Lagrangian of the problem. Among many applications, this paper proposes learning algorithms which extract adaptively reduced rank canonical variates and correlations, reduced rank Wiener filter, and principal and minor components within similar framework.