An optimization-oriented approach to the adaptive control of Markov chains

We consider the control of a dynamic system modeled as a Markov chain. The transition probability matrix of the Markov chain depends on the control u and also on an unknown parameter α. The unknown parameter belongs to a given finite set A. The long run average cost depends on the control policy and the unknown parameter. Thus a direct approach to the optimization of the performance is not feasible. A common procedure calls for an on-line estimation of the unknown parameter and the minimization of the cost functional using the estimate in lieu of the true parameter. It is well-known that this "certainty equivalence" (CE) solution may fail to yield the optimal performance. This motivates the presentation of a new optimiza tion-oriented approach to adaptive control. We consider a composite functional which simultaneously take care of theestimation and control needs. The global minimum of this composite functional coincides with the minimum of the original cost functional. Thus its joint minimization with respect to control and parameter estimates would yield the optimal control policy This joint minimization is not feasible, but it suggests an algorithm that asymptotically achieves the desired goal. Due to space constraints we omit a review of the literature as well as the proofs of our claims. They will be presented elsewhere.