A survey of dynamic spectrum allocation based on reinforcement learning algorithms in cognitive radio networks

Cognitive radio is an emerging technology that is considered to be an evolution for software device radio in which cognition and decision-making components are included. The main function of cognitive radio is to exploit “spectrum holes” or “white spaces” to address the challenge of the low utilization of radio resources. Dynamic spectrum allocation, whose significant functions are to ensure that cognitive users access the available frequency and bandwidth to communicate in an opportunistic manner and to minimize the interference between primary and secondary users, is a key mechanism in cognitive radio networks. Reinforcement learning, which rapidly analyzes the amount of data in a model-free manner, dramatically facilitates the performance of dynamic spectrum allocation in real application scenarios. This paper presents a survey on the state-of-the-art spectrum allocation algorithms based on reinforcement learning techniques in cognitive radio networks. The advantages and disadvantages of each algorithm are analyzed in their specific practical application scenarios. Finally, we discuss open issues in dynamic spectrum allocation that can be topics of future research.

[1]  Fangwen Fu,et al.  Detection of Spectral Resources in Cognitive Radios Using Reinforcement Learning , 2008, 2008 3rd IEEE Symposium on New Frontiers in Dynamic Spectrum Access Networks.

[2]  Kutluyil Dogançay,et al.  Distributed reduced-state SARSA algorithm for dynamic channel allocation in cellular networks featuring traffic mobility , 2005, IEEE International Conference on Communications, 2005. ICC 2005. 2005.

[3]  Sudharman K. Jayaweera,et al.  A Survey on Machine-Learning Techniques in Cognitive Radios , 2013, IEEE Communications Surveys & Tutorials.

[4]  Junaid Qadir,et al.  Artificial intelligence based cognitive routing for cognitive radio networks , 2013, Artificial Intelligence Review.

[5]  Hong Ji,et al.  Q-learning for dynamic channel assignment in cognitive wireless local area network with fibre-connected distributed antennas , 2012 .

[6]  Vijay K. Bhargava,et al.  Cognitive Wireless Communication Networks , 2007 .

[7]  Zhong Liang,et al.  A Survey on Distributed Opportunity Spectrum Access in Cognitive Network , 2010, 2010 6th International Conference on Wireless Communications Networking and Mobile Computing (WiCOM).

[8]  Sherali Zeadally,et al.  Spectrum Assignment in Cognitive Radio Networks: A Comprehensive Survey , 2013, IEEE Communications Surveys & Tutorials.

[9]  Rui Yang,et al.  Non-cooperative spectrum allocation based on game theory in cognitive radio networks , 2010, 2010 IEEE Fifth International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA).

[10]  Yong Zhang,et al.  Reinforcement-Learning-Based Double Auction Design for Dynamic Spectrum Access in Cognitive Radio Networks , 2013, Wirel. Pers. Commun..

[11]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[12]  Yan Huang,et al.  A factor graph based dynamic spectrum allocation approach for cognitive network , 2011, 2011 IEEE Wireless Communications and Networking Conference.

[13]  Vasileios Karyotis,et al.  A Markov Random Field framework for channel assignment in Cognitive Radio networks , 2012, 2012 IEEE International Conference on Pervasive Computing and Communications Workshops.

[14]  S. Haykin,et al.  A Q-learning-based dynamic channel assignment technique for mobile communication systems , 1999 .

[15]  Soamsiri Chantaraskul,et al.  Performance comparison of learning techniques for intelligent channel assignment in Cognitive Wireless Sensor Networks , 2015, 2015 Seventh International Conference on Ubiquitous and Future Networks.

[16]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[17]  Anjali Agarwal,et al.  Profit optimization in multi-service cognitive mesh network using machine learning , 2011, EURASIP J. Wirel. Commun. Netw..

[18]  Ayaz Ahmad,et al.  A Survey on Radio Resource Allocation in Cognitive Radio Sensor Networks , 2015, IEEE Communications Surveys & Tutorials.

[19]  Marco Levorato,et al.  A Learning Framework for Cognitive Interference Networks with Partial and Noisy Observations , 2012, IEEE Transactions on Wireless Communications.

[20]  Joseph Mitola,et al.  Cognitive Radio An Integrated Agent Architecture for Software Defined Radio , 2000 .

[21]  Mingyan Jiang,et al.  Cognitive radio spectrum assignment based on artificial bee colony algorithm , 2011, 2011 IEEE 13th International Conference on Communication Technology.

[22]  Kok-Lim Alvin Yau,et al.  Application of reinforcement learning to routing in distributed wireless networks: a review , 2013, Artificial Intelligence Review.

[23]  Ping Zhang,et al.  Radio resource management for public femtocell networks , 2011, EURASIP J. Wirel. Commun. Netw..

[24]  H. Vincent Poor,et al.  Reinforcement Learning-Based NOMA Power Allocation in the Presence of Smart Jamming , 2018, IEEE Transactions on Vehicular Technology.

[25]  Mei Song,et al.  Reinforcement Learning Based Auction Algorithm for Dynamic Spectrum Access in Cognitive Radio Networks , 2010, 2010 IEEE 72nd Vehicular Technology Conference - Fall.

[26]  Zhu Han,et al.  A Survey on Applications of Model-Free Strategy Learning in Cognitive Wireless Networks , 2015, IEEE Communications Surveys & Tutorials.

[27]  Ejaz Ahmed,et al.  Channel Assignment Algorithms in Cognitive Radio Networks: Taxonomy, Open Issues, and Challenges , 2016, IEEE Communications Surveys & Tutorials.

[28]  H. Vincent Poor,et al.  Two-dimensional anti-jamming communication based on deep reinforcement learning , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[29]  José Marinho,et al.  Cognitive radio: survey on communication protocols, spectrum decision issues, and future research directions , 2011, Wireless Networks.

[30]  Sixing Yin,et al.  Power and Spectrum Allocation in D2D Networks Based on Coloring and Chaos Genetic Algorithm , 2017 .

[31]  David Grace,et al.  Cognitive Radio with Reinforcement Learning Applied to Multicast Downlink Transmission with Power Adjustment , 2011, Wirel. Pers. Commun..

[32]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[33]  Lisandro Zambenedetti Granville,et al.  Improving reinforcement learning algorithms for dynamic spectrum allocation in cognitive sensor networks , 2013, 2013 IEEE Wireless Communications and Networking Conference (WCNC).

[34]  Hui Dai,et al.  A Q-learning-Based Dynamic Spectrum Allocation Algorithm , 2013 .

[35]  Bin Shen,et al.  Cooperative Spectrum Allocation in Centralized Cognitive Networks Using Bipartite Matching , 2008, IEEE GLOBECOM 2008 - 2008 IEEE Global Telecommunications Conference.

[36]  Joseph Mitola,et al.  Cognitive radio: making software radios more personal , 1999, IEEE Wirel. Commun..

[37]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[38]  Ping Zhang,et al.  Q-learning based heterogenous network self-optimization for reconfigurable network with CPC assistance , 2009, Science in China Series F: Information Sciences.

[39]  Brian M. Sadler,et al.  A Survey of Dynamic Spectrum Access , 2007, IEEE Signal Processing Magazine.

[40]  Haythem Bany Salameh,et al.  Throughput-oriented channel assignment for opportunistic spectrum access networks , 2011, Math. Comput. Model..

[41]  Kok-Lim Alvin Yau,et al.  Reinforcement learning for context awareness and intelligence in wireless networks: Review, new features and open issues , 2012, J. Netw. Comput. Appl..

[42]  Chonho Lee,et al.  Auction Approaches for Resource Allocation in Wireless Systems: A Survey , 2013, IEEE Communications Surveys & Tutorials.