Event-dependent and distributed Markov decision processes in communications

Many communication systems are distributed in nature where each geographically separated node has to make its own decisions. In addition, the actions in such systems depend on events, such as packet arrivals. In this paper, the authors study the optimization problem for such systems. Their method is based on the notion of potential. They show that by including the events into the states, the event-dependent systems can be modeled by the standard Markov decision process, and by introducing “aggregated potentials” the dimension of the problem can be reduced. They also propose a distributed approach to the problem, in which each node can estimate the “local potential”, and it only requires to transfer the estimated values of these local potentials among nodes, no state information has to be transferred