Second-order Markov reward models driven by QBD processes

Second-order reward models are an important class of models for evaluating the performance of real-life systems in which the reward measure fluctuates according to some underlying noise. These models consist of a Markov chain driving the evolution of the system, and a continuous reward variable representing its performance. Thus far, only models with a finite number of states have been studied. We consider second-order reward models driven by Quasi-birth-and-death processes, a class of block-structured Markov chains with infinitely many states. We derive the expressions for the Laplace–Stieltjes transforms of the accumulated reward and demonstrate how they can be efficiently evaluated. We use our results to analyse a simple example and, in doing so, show that the second-order feature can make a significant difference to the accumulated reward. The inclusion of the second-order feature also creates new difficulties which require the development of new conditions in the analysis.