A note on deterministic approximation of discounted Markov decision processes

We study the approximation of a small-noise Markov decision process x t = F ( x t − 1 , a t , ξ t ( ϵ ) ) , t = 1 , 2 , … by means of its deterministic counterpart: x ˜ t = F ( x ˜ t − 1 , a t , s 0 ) , t = 1 , 2 , … where s 0 is a fixed point of the disturbance metric space ( S , r ) . The total discounted cost is used as a criterion of optimality. Supposing that δ ϵ ≔ E r ( ξ 1 ( ϵ ) , s 0 ) → 0 as ϵ → 0 , we prove the convergence of optimal policies, estimate the rate of convergence of the optimal costs and give an upper bound (depending on δ ϵ ) for the stability index, which measures the excess of the cost due to a replacement of the optimal policy by its deterministic approximation.