Distributed Heuristically Accelerated Q-Learning for Robust Cognitive Spectrum Management in LTE Cellular Systems

In this paper, we propose an algorithm for dynamic spectrum access (DSA) in LTE cellular systems-distributed ICIC accelerated Q-learning (DIAQ). It combines distributed reinforcement learning (RL) and standardized inter-cell interference coordination (ICIC) signalling in the LTE downlink, using the framework of heuristically accelerated RL (HARL). Furthermore, we present a novel Bayesian network based approach to theoretical analysis of RL based DSA. It explains a predicted improvement in the convergence behaviour achieved by DIAQ, compared to classical RL. The scheme is also assessed using large scale simulations of a stadium temporary event network. Compared to a typical heuristic ICIC approach, DIAQ provides significantly better quality of service and supports considerably higher network throughput densities. In addition, DIAQ dramatically improves initial performance, speeds up convergence, and improves steady state performance of a state-of-the-art distributed Q-learning algorithm, confirming the theoretical predictions. Finally, our scheme is designed to comply with the current LTE standards. Therefore, it enables easy implementation of robust distributed machine intelligence for full self-organisation in existing commercial networks.

[1]  Reinaldo A. C. Bianchi,et al.  Case-Based Multiagent Reinforcement Learning: Cases as Heuristics for Selection of Actions , 2010, ECAI.

[2]  David Grace,et al.  Efficient exploration in reinforcement learning-based cognitive radio spectrum sharing , 2011, IET Commun..

[3]  Karina Mabell Gomez,et al.  Airborne Base Stations for Emergency and Temporary Events , 2013, PSATS.

[4]  Tim Clarke,et al.  Distributed Q-learning based dynamic spectrum management in cognitive cellular systems: Choosing the right learning rate , 2014, 2014 IEEE Symposium on Computers and Communications (ISCC).

[5]  Zhu Han,et al.  Self-Organization in Small Cell Networks: A Reinforcement Learning Approach , 2013, IEEE Transactions on Wireless Communications.

[6]  Reinaldo A. C. Bianchi,et al.  Heuristically-Accelerated Multiagent Reinforcement Learning , 2014, IEEE Transactions on Cybernetics.

[7]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[8]  Stefania Sesia,et al.  LTE - The UMTS Long Term Evolution, Second Edition , 2011 .

[9]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[10]  Manuela M. Veloso,et al.  Multiagent learning using a variable learning rate , 2002, Artif. Intell..

[11]  Tim Clarke,et al.  Transfer Learning: A Paradigm for Dynamic Spectrum and Topology Management in Flexible Architectures , 2013, 2013 IEEE 78th Vehicular Technology Conference (VTC Fall).

[12]  Panagiotis Demestichas,et al.  Autonomic downlink inter-cell interference coordination in LTE Self-Organizing Networks , 2011, 2011 7th International Conference on Network and Service Management.

[13]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[14]  Stavros A. Kotsopoulos,et al.  A Decentralized Subchannel Allocation Scheme with Inter-Cell Interference Coordination (ICIC) for Multi-Cell OFDMA Systems , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[15]  Rui Chang,et al.  Interference coordination and cancellation for 4G networks , 2009, IEEE Communications Magazine.

[16]  S. Haykin,et al.  A Q-learning-based dynamic channel assignment technique for mobile communication systems , 1999 .

[17]  Meryem Simsek,et al.  Dynamic Inter-Cell Interference Coordination in HetNets: A reinforcement learning approach , 2012, 2012 IEEE Global Communications Conference (GLOBECOM).

[18]  Xianfu Chen,et al.  Stochastic Power Adaptation with Multiagent Reinforcement Learning for Cognitive Wireless Mesh Networks , 2013, IEEE Transactions on Mobile Computing.

[19]  David Grace,et al.  Case-based reinforcement learning for cognitive spectrum assignment in cellular networks with dynamic topologies , 2013, 2013 Military Communications and Information Systems Conference.

[20]  Jungwon Lee,et al.  Advanced interference management for 5G cellular networks , 2014, IEEE Communications Magazine.

[21]  Zwi Altman,et al.  A cooperative Reinforcement Learning approach for Inter-Cell Interference Coordination in OFDMA cellular networks , 2010, 8th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks.

[22]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.