Sufficiency of Markov policies for continuous-time Markov decision processes and solutions to Kolmogorov´s forward equation for jump Markov processes

In continuous-time Markov decision processes (CTMDPs) with Borel state and action spaces, unbounded transition rates, for an arbitrary policy, we construct a relaxed Markov policy such that the marginal distribution on the state-action pairs at any time instant is the same for both the policies. This result implies the existence of a relaxed Markov policy that performs equally to an arbitrary policy with respect to expected discounted and non-discounted total costs as well as average costs per unit time. The proof consists of two steps. The first step describes the properties of solutions to Kolmogorov´s forward equation for jump Markov Processes. The second step applies these results to CTMDPs.