A cooperative Q-learning approach for distributed resource allocation in multi-user femtocell networks

This paper studies distributed interference management for femtocells that share the same frequency band with macrocells. We propose a multi-agent learning technique based on distributed Q-learning, called subcarrier-based distributed resource allocation using Q-learning (SBDRA-Q). SBDRA-Q operates under three different learning paradigms: Independent (IL), Cooperative (CL) and Weighted Cooperative (WCL). In the IL paradigm, all femtocells learn independently from each other. In both, CL and WCL, femtocells share partial information during the learning process in order to enhance their performance. The results show that WCL outperforms both CL and IL in terms of aggregate femtocell capacity, while slightly affecting fairness. Also, the results show that CL and WCL are more robust, when compared to IL, to new femtocells being deployed during the learning process. Finally, we show SBDRA-Q achieves higher aggregate femtocell capacity under the three learning paradigms when compared to a power allocation scheme (SBDPC-Q) that was proposed in the literature.