Existence of an optimal control for systems with jump Markov disturbances

We consider stochastic optimal control problems of Mayer type with dynamics in the form of a system of ordinary differential equations perturbed by a countable state Markov process, and we prove the existence of an optimal control in the class of non-anticipative functions. The proof takes the same approach as the "direct" method of the calculus of variations, used extensively in deterministic problems, but substitutes probabilistic concepts where necessary.