Title :
Learning control of finite Markov chains with an explicit trade-off between estimation and control
Author :
Sato, Mitsuo ; Abe, Kenichi ; Takeda, Hiroshi
Author_Institution :
Dept. of Electr. Eng., Tohoku Univ., Sendai, Japan
Abstract :
An efficient scheme is presented for a learning control problem of finite Markov chains with unknown dynamics, i.e. with unknown transition probabilities. The scheme is designed to optimize the asymptotic system performance and for easy application to models with relatively many states and decisions. In this scheme a control policy is determined each time through maximization of a simple performance criterion that explicitly incorporates a tradeoff between estimation of the unknown probabilities and control of the system. The policy determination can be easily performed even in the case of large-size models, since the maximizing operation can be greatly simplified by use of the policy-iteration method. It is proven that this scheme becomes ε-optimal as well as optimal by suitable choice of control parameter values in the sense that a relative frequency coefficient of making optimal decisions tends to the maximum
Keywords :
Markov processes; adaptive control; decision theory; learning systems; optimisation; probability; adaptive control; decision theory; finite Markov chains; learning control; maximization; optimisation; policy-iteration method; transition probabilities; Automatic control; Bayesian methods; Control systems; Convergence; Design optimization; Frequency; Learning automata; Optimal control; Stochastic systems; System performance;
Journal_Title :
Systems, Man and Cybernetics, IEEE Transactions on