Dynamic Spectrum Sharing Based on Deep Reinforcement Learning in Mobile Communication Systems

The rapid development of mobile communication services in recent years has resulted in a scarcity of spectrum resources. This paper addresses the problem of multi-dimensional resource allocation in cognitive radio systems. Deep reinforcement learning (DRL) combines deep learning and reinforcement learning to enable agents to solve complex problems. In this study, we propose a training approach based on DRL to design a strategy for secondary users in the communication system to share the spectrum and control their transmission power. The neural networks are constructed using the Deep Q-Network and Deep Recurrent Q-Network structures. The results of the conducted simulation experiments demonstrate that the proposed method can effectively improve the user’s reward and reduce collisions. In terms of reward, the proposed method outperforms opportunistic multichannel ALOHA by about 10% and about 30% for the single SU scenario and the multi-SU scenario, respectively. Furthermore, we explore the complexity of the algorithm and the influence of parameters in the DRL algorithm on the training.

[1]  Yusheng Ji,et al.  A Usage Aware Dynamic Spectrum Access Scheme for Interweave Cognitive Radio Network by Exploiting Deep Reinforcement Learning , 2022, Sensors.

[2]  Xin Liu,et al.  Reinforcement-Learning-Based Dynamic Spectrum Access for Software-Defined Cognitive Industrial Internet of Things , 2022, IEEE Transactions on Industrial Informatics.

[3]  Jiaying Zong,et al.  6G Cell-Free Network Architecture , 2022, 2022 IEEE 2nd International Conference on Electronic Technology, Communication and Information (ICETCI).

[4]  Bin Shen,et al.  Deep Q-network Based Reinforcement Learning for Distributed Dynamic Spectrum Access , 2022, 2022 IEEE 2nd International Conference on Computer Communication and Artificial Intelligence (CCAI).

[5]  Yeong-Chan Kim,et al.  Deep Reinforcement Learning for Dynamic Spectrum Access in the Multi-Channel Wireless Local Area Networks , 2022, 2022 International Conference on Electronics, Information, and Communication (ICEIC).

[6]  Jian Song,et al.  Deep Reinforcement Learning for Spectrum Sharing in Future Mobile Communication System , 2021, 2021 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB).

[7]  Lingjia Liu,et al.  A Deep Reinforcement Learning Framework for Spectrum Management in Dynamic Spectrum Access , 2021, IEEE Internet of Things Journal.

[8]  Wenhui Lang,et al.  Deep Multi-User Reinforcement Learning for Centralized Dynamic Multichannel Access , 2021, 2021 6th International Conference on Intelligent Computing and Signal Processing (ICSP).

[9]  Weiheng Jiang,et al.  Multi-Agent Reinforcement Learning for Joint Cooperative Spectrum Sensing and Channel Access in Cognitive UAV Networks , 2021, Sensors.

[10]  Pin Wan,et al.  Intelligent Dynamic Spectrum Access Using Deep Reinforcement Learning for VANETs , 2021, IEEE Sensors Journal.

[11]  Vahid Pourahmadi,et al.  Deep-Reinforcement Learning for Fair Distributed Dynamic Spectrum Access in Wireless Networks , 2021, 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC).

[12]  Lin Qi,et al.  A New Distributed Dynamic Spectrum Access Model Based on DQN , 2020, 2020 15th IEEE International Conference on Signal Processing (ICSP).

[13]  Yibing Li,et al.  Power Control Based on Deep Q Network with Modified Reward Function in Cognitive Networks , 2020, 2020 IEEE USNC-CNC-URSI North American Radio Science Meeting (Joint with AP-S Symposium).

[14]  Y. Xu,et al.  The Application of Deep Reinforcement Learning to Distributed Spectrum Access in Dynamic Heterogeneous Environments With Partial Observations , 2020, IEEE Transactions on Wireless Communications.

[15]  Victor C. M. Leung,et al.  Power Control Based on Deep Reinforcement Learning for Spectrum Sharing , 2020, IEEE Transactions on Wireless Communications.

[16]  Yu Liu,et al.  Deep Reinforcement Learning for Dynamic Spectrum Sensing and Aggregation in Multi-Channel Wireless Networks , 2020, IEEE Transactions on Cognitive Communications and Networking.

[17]  NaiQian Zhang,et al.  An Advanced Algorithm for Spectrum Allocation of Primary Users Based on Cournot Game , 2019, 2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP).

[18]  Y. Xu,et al.  Cache-Enabled Dynamic Spectrum Access via Deep Recurrent Q-Networks with Partial Observation , 2019, 2019 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN).

[19]  Francois Gagnon,et al.  Quantifying range extension capability of MIMO: a simulation study based on WINNER II model , 2019, 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE).

[20]  Haibo He,et al.  Distributive Dynamic Spectrum Access Through Deep Reinforcement Learning: A Reservoir Computing-Based Approach , 2018, IEEE Internet of Things Journal.

[21]  Ying-Chang Liang,et al.  Applications of Deep Reinforcement Learning in Communications and Networking: A Survey , 2018, IEEE Communications Surveys & Tutorials.

[22]  R. Michael Buehrer,et al.  Deep Reinforcement Learning for Dynamic Spectrum Access in Wireless Networks , 2018, MILCOM 2018 - 2018 IEEE Military Communications Conference (MILCOM).

[23]  Tuan A. Nguyen,et al.  Deep Q-Learning with Multiband Sensing for Dynamic Spectrum Access , 2018, 2018 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN).

[24]  R. Michael Buehrer,et al.  Dealing with Partial Observations in Dynamic Spectrum Access: Deep Recurrent Q-Networks , 2018, MILCOM 2018 - 2018 IEEE Military Communications Conference (MILCOM).

[25]  Wei Liang,et al.  Dynamic Spectrum Access in Cognitive Radio Networks Using Deep Reinforcement Learning and Evolutionary Game , 2018, 2018 IEEE/CIC International Conference on Communications in China (ICCC).

[26]  Bhaskar Krishnamachari,et al.  Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks , 2018, IEEE Transactions on Cognitive Communications and Networking.

[27]  Zhi Chen,et al.  Intelligent Power Control for Spectrum Sharing in Cognitive Radios: A Deep Reinforcement Learning Approach , 2017, IEEE Access.

[28]  M. Y. Alias,et al.  Efficient handoff spectrum scheme using fuzzy decision making in cognitive radio system , 2017, 2017 3rd International Conference on Frontiers of Signal Processing (ICFSP).

[29]  Kobi Cohen,et al.  Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access , 2017, IEEE Transactions on Wireless Communications.

[30]  Abdulraqeb Alhammadi,et al.  Fuzzy logic based negotiation approach for spectrum handoff in cognitive radio network , 2016, 2016 IEEE 3rd International Symposium on Telecommunication Technologies (ISTT).

[31]  Abdulraqeb Alhammadi,et al.  Analysis of spectrum handoff schemes in cognitive radio network using particle swarm optimization , 2016, 2016 IEEE 3rd International Symposium on Telecommunication Technologies (ISTT).

[32]  Yuehong Gao,et al.  Spectrum sharing for LTE and WiFi coexistence using decision tree and game theory , 2016, 2016 IEEE Wireless Communications and Networking Conference.

[33]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[34]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[35]  Ping Zhang,et al.  Joint price and power allocation under interference constraint for dynamic spectrum access networks , 2014, 2014 IEEE International Symposium on Dynamic Spectrum Access Networks (DYSPAN).

[36]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[37]  G. Singh,et al.  Spectrum sharing in Cognitive Radio using game theory , 2013, 2013 3rd IEEE International Advance Computing Conference (IACC).

[38]  A. De,et al.  Online dynamic resource allocation in interference temperature constrained cognitive radio network using reinforcement learning , 2012, 2012 International Conference on Signal Processing and Communications (SPCOM).

[39]  Qi Zhu,et al.  An Improved Game-Theoretic Algorithm for Competitive Spectrum Sharing , 2010, 2010 International Conference on Communications and Mobile Computing.

[40]  Martin Döttling,et al.  Radio technologies and concepts for IMT-Advanced , 2009 .

[41]  Lassi Hentila,et al.  WINNER II Channel Models , 2009 .

[42]  Simon Haykin,et al.  Cognitive radio: brain-empowered wireless communications , 2005, IEEE Journal on Selected Areas in Communications.

[43]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[44]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[45]  Jinho Choi,et al.  On exploiting idle channels in opportunistic multichannel ALOHA , 2010, IEEE Communications Letters.

[46]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 2005, IEEE Transactions on Neural Networks.