The Poisson equation for reversible Markov chains: Analysis and application to Markov chain samplers

The steady-state behavior a finite-state Markov chain can be evaluated by solving a Poisson equation, a special case of the optimality equation arising in average cost Markov decision processes. In practice, solving the Poisson equation is typically no easier than evaluating steady state behavior directly using the invariant probability mass function of the Markov chain. However, it is known that approximate solutions to the Poisson equation can be used to produce bounds on steady state performance and to accelerate simulations for estimating steady state behavior. In this paper we study the special structure taken on by the Poisson equation when the associated Markov chain is reversible. In particular, we show that reversible Markov chains have a special form of the Poisson equation that admits a closed form solution. As an application of the reversible Poisson equation we consider the construction of control variates that can be used in Markov chain-based samplers. We show one class of control variates that are obtained by approximating the known solution to the reversible Poisson equation, and demonstrate the application of these control variates on an example involving the Metropolis-Hastings algorithm.