Reinforcement control via action dependent heuristic dynamic programming

Heuristic dynamic programming (HDP) is the simplest kind of adaptive critic which is a powerful form of reinforcement control. It can be used to maximize or minimize any utility function, such as total energy or trajectory error, of a system over time in a noisy environment. Unlike supervised learning, adaptive critic design does not require the desired control signals be known. Instead, feedback is obtained based on a critic network which learns the relationship between a set of control signals and the corresponding strategic utility function. It is an approximation of dynamic programming. Action-dependent heuristic dynamic programming (ADHDP) system involves two subnetworks, the action network and the critic network. Each of these networks includes a feedforward and a feedback component. A flow chart for the interaction of these components is included. To further illustrate the algorithm, we use ADHDP for the control of a simple, 2D planar robot