Delay-optimal power and subcarrier allocation for OFDMA systems via stochastic approximation

In this paper, we consider delay-optimal power and subcarrier allocation design for OFDMA systems with N_F subcarriers, K mobiles and one base station. There are K queues at the base station for the downlink traffic to the K mobiles with heterogeneous packet arrivals and delay requirements. We shall model the problem as a K-dimensional infinite horizon average reward Markov Decision Problem (MDP) where the control actions are assumed to be a function of the instantaneous Channel State Information (CSI) as well as the joint Queue State Information (QSI). We propose an online stochastic value iteration solution using stochastic approximation. The proposed power control algorithm, which is a function of both the CSI and the QSI, takes the form of multi-level water-filling. We prove that under two mild conditions in Theorem 1, the proposed solution converges to the optimal solution almost surely (with probability 1) and the proposed framework offers a possible solution to the general stochastic NUM problem. By exploiting the birth-death structure of the queue dynamics, we obtain a reduced complexity decomposed solution with linear O(KN_F) complexity and O(K) memory requirement.