Computationally efficient adaptive control algorithms for Markov chains

Algorithms for adaptive control of unknown finite Markov chains are proposed. The algorithms consist of two parts: part one estimates the unknown parameters; part two computes the optimal policy. In this study the emphasis is on efficient online computation of the optimal policy. No a priori knowledge of the optimal policy is assumed. The optimal policy is computed recursively online. At each step a small amount of computation is required. At each transition of the chain, only the act corresponding to the present state of the chain is updated. The algorithms are easy to implement and converge to the optimal policy in finite time