Monotonicity of Constrained Optimal Transmission Policies in Correlated Fading Channels With ARQ

We consider transmission scheduling using an ARQ protocol with retransmissions given channel state information (CSI) and a correlated fading channel. The problem is formulated as a countable state, infinite horizon, average cost Markov decision process (MDP) with an average delay constraint. Our main result is to give sufficient conditions on the channel memory, and transmission cost so that the optimal transmission scheduling policy is a monotonically increasing function of the buffer occupancy. In proving this result, we first prove positive recurrence (stability) of the buffer. The monotone structure proof consists of two steps. First, the constrained MDP (CMDP) is transformed into an unconstrained MDP using a Lagrangian dynamic programming formulation. It is proved that the unconstrained optimal policy is pure and monotonically increasing in the buffer occupancy. It is then shown that the constrained optimal policy is a randomized mixture of two pure transmission policies that are monotone in the buffer state. Finally, the monotone structure of the optimal transmission policy is exploited to derive a monotone-policy Q -learning algorithm and a stochastic approximation based monotone policy search algorithm for estimating the optimal policy in real time.