Model-free Approximate Dynamic Programming Schemes for Linear Systems

In this paper, we present online model-free adaptive critic (AC) schemes based on approximate dynamic programming (ADP) to solve optimal control problems in both discrete-time and continuous-time domains for linear systems with unknown dynamics. In the discrete-time case, it is shown that the proposed ADP algorithm is in fact solving the underlying Generalized Algebraic Riccati Equation (GARE) of the corresponding optimal control problem or zero-sum game. In the continuous-time domain, an ADP scheme is introduced to solve for the underlying ARE of the optimal control problem. It is shown that this continuous-time ADP scheme is in fact a Quasi-Newton method to solve the ARE. In both time domains, the adaptive critic algorithms are easy to initialize since initial policies are not required to be stabilizing. It is also shown, on a power system control example, that both discrete-time and continuous-time approaches to ADP converge to the same continuous time optimal control solution provided that the utility function is appropriately chosen.