Cognitive spectrum management in dynamic cellular environments: A case-based Q-learning approach

This paper examines how novel cellular system architectures and intelligent spectrum management techniques can be used to play a key role in accommodating the exponentially increasing demand for mobile data capacity in the near future. A significant challenge faced by the artificial intelligence methods applied to such flexible wireless communication systems is their dynamic nature, e.g. network topologies that change over time. This paper proposes an intelligent case-based Q-learning method for dynamic spectrum access (DSA) which improves and stabilises the performance of cognitive cellular systems with dynamic topologies. The proposed approach is the combination of classical distributed Q-learning and a novel implementation of case-based reasoning which aims to facilitate a number of learning processes running in parallel. Large scale simulations of a stadium small cell network show that the proposed case-based Q-learning approach achieves a consistent improvement in the system quality of service (QoS) under dynamic and asymmetric network topology and traffic load conditions. Simulations of a secondary spectrum sharing scenario show that the cognitive cellular system that employs the proposed case-based Q-learning DSA scheme is able to accommodate a 28-fold increase in the total primary and secondary system throughput, but with no need for additional spectrum and with no degradation in the primary user QoS.

[1]  Tim Clarke,et al.  Distributed Heuristically Accelerated Q-Learning for Robust Cognitive Spectrum Management in LTE Cellular Systems , 2016, IEEE Transactions on Mobile Computing.

[2]  Cheng-Xiang Wang,et al.  Wideband spectrum sensing for cognitive radio networks: a survey , 2013, IEEE Wireless Communications.

[3]  Stefania Sesia,et al.  LTE - The UMTS Long Term Evolution, Second Edition , 2011 .

[4]  George T. Karetsos,et al.  IST-4-027756 WINNER II Deliverable D6.11.4 Final WINNER II System Requirements , 2007 .

[5]  Sami Tabbane,et al.  Win-win relationship between macrocell and femtocells for spectrum sharing in LTE-A , 2014, IET Commun..

[6]  Tim Clarke,et al.  Heuristically Accelerated Reinforcement Learning for Dynamic Secondary Spectrum Sharing , 2015, IEEE Access.

[7]  Xianfu Chen,et al.  Stochastic Power Adaptation with Multiagent Reinforcement Learning for Cognitive Wireless Mesh Networks , 2013, IEEE Transactions on Mobile Computing.

[8]  Reinaldo A. C. Bianchi,et al.  Reinforcement Learning with Case-Based Heuristics for RoboCup Soccer Keepaway , 2012, 2012 Brazilian Robotics Symposium and Latin American Robotics Symposium.

[9]  Ethan S. Hennessey,et al.  An Architecture for Coexistence with Multiple Users in Frequency Hopping Cognitive Radio Networks , 2014, IEEE Journal on Selected Areas in Communications.

[10]  Klaus Moessner,et al.  Dynamic Heterogeneous Learning Games for Opportunistic Access in LTE-Based Macro/Femtocell Deployments , 2015, IEEE Transactions on Wireless Communications.

[11]  L. Chiaraviglio,et al.  Optimal Energy Savings in Cellular Access Networks , 2009, 2009 IEEE International Conference on Communications Workshops.

[12]  Zhu Han,et al.  Self-Organization in Small Cell Networks: A Reinforcement Learning Approach , 2013, IEEE Transactions on Wireless Communications.

[13]  Mohsen Guizani,et al.  Large-scale cognitive cellular systems: resource management overview , 2015, IEEE Communications Magazine.

[14]  Cheng-Xiang Wang,et al.  Wideband Spectrum Sensing for Cognitive Radio Networks , 2013, ArXiv.

[15]  Salahedin Rehan,et al.  Aerial base stations with opportunistic links for next generation emergency communications , 2016, IEEE Communications Magazine.

[16]  Dave Cavalcanti,et al.  Coexistence challenges for heterogeneous cognitive wireless networks in TV white spaces , 2011, IEEE Wireless Communications.

[17]  Daniel Kudenko,et al.  Distributed response to network intrusions using multiagent reinforcement learning , 2015, Eng. Appl. Artif. Intell..

[18]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[19]  Tim Clarke,et al.  Distributed Q-learning based dynamic spectrum access in high capacity density cognitive cellular systems using secondary LTE spectrum sharing , 2014, 2014 International Symposium on Wireless Personal Multimedia Communications (WPMC).

[20]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[21]  Petri Ahokangas,et al.  Spectrum sharing using licensed shared access: the concept and its workflow for LTE-advanced networks , 2014, IEEE Wireless Communications.

[22]  Jin Ma,et al.  An integrated feature selection and cluster analysis techniques for case-based reasoning , 2015, Eng. Appl. Artif. Intell..

[23]  Hossein Nezamabadi-pour,et al.  Long term learning in image retrieval systems using case based reasoning , 2014, Eng. Appl. Artif. Intell..

[24]  Zhaohan Sheng,et al.  Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system , 2009, Expert Syst. Appl..

[25]  David Grace,et al.  Efficient exploration in reinforcement learning-based cognitive radio spectrum sharing , 2011, IET Commun..

[26]  Tim Clarke,et al.  Distributed Q-learning based dynamic spectrum management in cognitive cellular systems: Choosing the right learning rate , 2014, 2014 IEEE Symposium on Computers and Communications (ISCC).

[27]  Stavros A. Kotsopoulos,et al.  A Decentralized Subchannel Allocation Scheme with Inter-Cell Interference Coordination (ICIC) for Multi-Cell OFDMA Systems , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[28]  A. Goldsmith,et al.  Cognitive Cellular Systems within the TV Spectrum , 2010, 2010 IEEE Symposium on New Frontiers in Dynamic Spectrum (DySPAN).

[29]  David Grace,et al.  Application of reinforcement learning on medium access control for wireless sensor networks , 2013 .

[30]  Matthijs T. J. Spaan,et al.  Traffic flow optimization: A reinforcement learning approach , 2016, Eng. Appl. Artif. Intell..

[31]  Manuela M. Veloso,et al.  Multiagent learning using a variable learning rate , 2002, Artif. Intell..

[32]  Gerhard Fettweis,et al.  Energy Efficiency Aspects of Base Station Deployment Strategies for Cellular Networks , 2009, 2009 IEEE 70th Vehicular Technology Conference Fall.

[33]  David Grace,et al.  Case-based reinforcement learning for cognitive spectrum assignment in cellular networks with dynamic topologies , 2013, 2013 Military Communications and Information Systems Conference.

[34]  J. Grosspietsch,et al.  Geo-Location Database Techniques for Incumbent Protection in the TV White Space , 2008, 2008 3rd IEEE Symposium on New Frontiers in Dynamic Spectrum Access Networks.

[35]  Reinaldo A. C. Bianchi,et al.  Transferring knowledge as heuristics in reinforcement learning: A case-based approach , 2015, Artif. Intell..

[36]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[37]  Salahedin Rehan,et al.  Combined green resource and topology management for beyond next generation mobile broadband systems , 2013, 2013 International Conference on Computing, Networking and Communications (ICNC).