Adaptive QoS provisioning in wireless ad hoc networks: a semi-MDP approach

The paper presents a joint bandwidth allocation and buffer management scheme for QoS provisioning in a differentiated services framework on a wireless ad hoc network. Our proposed scheme models the system as a semi-Markov decision process (SMDP) and uses a novel model-free average reward reinforcement learning (RL) algorithm that considers maximizing the average long term reward for our network and, at the same time, minimizing QoS violations with respect to bandwidth, queueing delay and buffer loss. Due to the nature of the provisioning problem of having continuous and multi-dimensional state and action spaces, we also present a novel function approximation technique (wire-fitted CMAC) that uses a linear tile-coding structure together with a wire-fitted interpolation. Using a linear approximator facilitates the convergence of our proposed RL algorithm. The wire-fitted CMAC also generalizes both the state and action values. Simulation results show the effectiveness and convergence of the proposed provisioning scheme with respect to the average long term reward.