Online learning algorithm for Stackelberg games in problems with hierarchy

This paper presents an online adaptive optimal control algorithm based on policy iteration reinforcement learning techniques to solve the continuous-time Stackelberg games with infinite horizon for linear systems. This adaptive optimal control method finds in real-time approximations of the optimal value and the Stackelberg-equilibrium solution, while also guaranteeing closed-loop stability. The optimal-adaptive algorithm is implemented as a separate actor/critic parametric network approximator structure for every player, and involves simultaneous continuous-time adaptation of the actor/critic networks. Novel tuning algorithms are given for the actor/critic networks. The convergence to the closed-loop Stackelberg equilibrium is proven and stability of the system is also guaranteed. A simulation example shows the effectiveness of the new online algorithm.