Model-free adaptive algorithm for optimal control of continuous-time nonlinear system

Reinforcement learning has provided an efficient approach to solve the optimal control of some complicated systems. In this paper we resort to the idea of off-policy scheme and propose a complete-model-free algorithm to the continuous-time nonlinear optimal control problems. The novel algorithm consists of two neural networks to approximate the value and policy and adapts them with continuous tuning laws. In addition, experience replay technique is employed so that both the instantaneous observations and the past data are utilized. The convergence to the optimal solutions are guaranteed under the Lyapunov analysis. A nonlinear system is simulated to test the learning performance.