Determining Near-Optimal Policies for Markov Renewal Decision Processes

Two types of recurrence equations arise in finite-horizon Markov renewal decision processes. One gives expected return when the goal is to maximize return per transition (transition optimal); the other gives expected return when the goal is to maximize return per unit time (time optimal). Previous work [1]-[3] has shown how the transition-optimal equations can solve the infinite-horizon problem by providing near-optimal policies and bounds on the gains of these policies and the optimal policies. This correspondence obtains analogous results with the time-optimal equations.