Constructing performance sensitivities of Markov systems with potentials as building blocks

We study the structure of sample paths of Markov systems by using performance potentials as the fundamental units. With a sample path-based approach, we show that performance sensitivities of Markov systems can be constructed by using performance potentials (or equivalently, perturbation realization factors) as building blocks. We propose an intuitive approach to derive, by first principles, formulas for performance derivatives and performance differences for two Markov chains. These formulas are the basis for performance optimization of discrete event dynamic systems, including perturbation analysis, Markov decision processes, and reinforcement learning.