Self-evaluating reinforcement learning based spectrum management for cognitive Ad Hoc networks

Cognitive radio (CR) is an emerging technology which aims to solve the current wireless spectra problems in terms of efficiency and utilization. CRs should respond adequately to environmental changes in order to operate in a highly efficient manner. Reinforcement learning (RL) is an effective method that entails exploration, followed by exploitation and is used to train CRs to function in unknown environments. However, it also suffers from cases that cannot be easily avoided and recovered from, such as: conservative behavior causing converging to a non-ideal state and aggressive exploration that results in disrupting the network. In this paper, we propose a self-evaluating, RL-based spectrum management approach for cognitive ad-hoc networks. We investigate a means to detect environmental changes by having a CR inspect its information consistency and respond accordingly to changes in the environment. We also aim to grant CRs more flexibility in exploration behavior since using this approach will make it easier to remedy any shortcomings caused by aggressive exploration. The benefit of applying our algorithm is demonstrated and comparisons of performances using evaluations of different scopes are also provided to illustrate their impact on the spectrum management. Simulation results show the proposed approach is effective and able to improve the performance by increasing CRs´ responsiveness to environmental changes and allowing fast recovery.