Delay-Optimal Resource Allocation for OFDMA Systems via Stochastic Approximation

In this paper, we consider delay-optimal power and subcarrier allocation design for OFDMA systems with K mobiles and one base station. There are K queues at the base station for the downlink traffic to the K mobiles with heterogeneous packet arrivals and delay requirements. We shall model the problem as a K-dimensional infinite horizon average reward Markov decision problem (MDP) where the control actions are assumed to be a function of the instantaneous channel state information (CSI) as well as the joint queue state information (QSI). This problem is challenging because it corresponds to a stochastic network utility maximization (NUM) problem where general solution is still unknown. We propose an online stochastic value iteration solution using stochastic approximation. The proposed power control algorithm, which is a function of both the CSI and the QSI, takes the form of multi-level water-filling. We prove that under some mild conditions, the proposed solutions converge to the optimal solution almost surely and the proposed framework offers a possible solution to the general stochastic NUM problem. By exploiting the birth-death structure of the queue dynamics in the Poisson arrivals, we obtain a reduced complexity decomposed solution with linear O(KN_F) complexity and memory requirement.