Analysis of an adaptive control scheme for a partially observed controlled Markov chain

The authors present parameter estimation techniques based on the information available after actions that reset the state to a known value are taken. At these times, the (augmented) state process `regenerates,´ its future evolution becoming independent of the past. Using the ordinary differential equation method, w.p.1 convergence of the parameter estimates to the true (unknown) parameter θ₀ is demonstrated for two parameter estimation schemes of this type. Then, given any sequence of parameter estimates which is measurable with respect to the filtration generated by the observations and converges w.p.1 to θ₀, the expected long-run average cost incurred by the certainty equivalent adaptive policy is the same as that incurred by the optimal stationary policy corresponding to the true parameter. The latter is obtained by a detailed analysis which uses the known parameters