Deep Reinforcement Learning for Dynamic Spectrum Sensing and Aggregation in Multi-Channel Wireless Networks

In this paper, the problem of dynamic spectrum sensing and aggregation is investigated in a wireless network containing <inline-formula> <tex-math notation="LaTeX">${N}$ </tex-math></inline-formula> correlated channels, where these channels are occupied or vacant following an unknown joint 2-state Markov model. At each time slot, a single cognitive user with certain bandwidth requirement either stays idle or selects a segment comprising <inline-formula> <tex-math notation="LaTeX">${C}$ </tex-math></inline-formula> (<inline-formula> <tex-math notation="LaTeX">${C} < {N}$ </tex-math></inline-formula>) continuous channels to sense. Then, the vacant channels in the selected segment will be aggregated for satisfying the user requirement. The user receives a binary feedback signal indicating whether the transmission is successful or not (i.e., ACK signal) after each transmission, and makes next decision based on the sensing channel states. Here, we aim to find a policy that can maximize the number of successful transmissions without interrupting the primary users (PUs). The problem can be considered as a partially observable Markov decision process (POMDP) due to without full observation of system environment. We implement a Deep Q-Network (DQN) to address the challenge of unknown system dynamics and computational expenses. The performance of DQN, Q-Learning, and the Improvident Policy with known system dynamics is evaluated through simulations. The simulation results show that DQN can achieve near-optimal performance among different system scenarios only based on partial observations and ACK signals.

[1]  Alagan Anpalagan,et al.  Anti-Jamming Communications Using Spectrum Waterfall: A Deep Reinforcement Learning Approach , 2017, IEEE Communications Letters.

[2]  Zhisheng Niu,et al.  DeepNap: Data-Driven Base Station Sleeping Operations Through Deep Reinforcement Learning , 2018, IEEE Internet of Things Journal.

[3]  Betim Çiço,et al.  Computer vision approaches based on deep learning and neural networks: Deep neural networks for video analysis of human pose estimation , 2017, 2017 6th Mediterranean Conference on Embedded Computing (MECO).

[4]  Hongyang Chen,et al.  Double Coded Caching in Ultra Dense Networks: Caching and Multicast Scheduling via Deep Reinforcement Learning , 2020, IEEE Transactions on Communications.

[5]  Tuan A. Nguyen,et al.  Deep Q-Learning with Multiband Sensing for Dynamic Spectrum Access , 2018, 2018 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN).

[6]  Jie Xu,et al.  Data-driven dynamic resource scheduling for network slicing: A Deep reinforcement learning approach , 2019, Inf. Sci..

[7]  Aggelos K. Katsaggelos,et al.  Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods , 2018, IEEE Signal Processing Magazine.

[8]  Guanding Yu,et al.  Prediction-Based Spectrum Aggregation with Hardware Limitation in Cognitive Radio Networks , 2010, 2010 IEEE 71st Vehicular Technology Conference.

[9]  Cheng-Xiang Wang,et al.  Enhanced 5G Cognitive Radio Networks Based on Spectrum Sharing and Spectrum Aggregation , 2018, IEEE Transactions on Communications.

[10]  Jie Huang,et al.  A Big Data Enabled Channel Model for 5G Wireless Communication Systems , 2020, IEEE Transactions on Big Data.

[11]  Qian Xu,et al.  Predicting Wireless MmWave Massive MIMO Channel Characteristics Using Machine Learning Algorithms , 2018, Wirel. Commun. Mob. Comput..

[12]  Miguel López-Benítez,et al.  An Overview of Spectrum Occupancy Models for Cognitive Radio Networks , 2011, Networking Workshops.

[13]  Yasir Saleem,et al.  Primary radio user activity models for cognitive radio networks: A survey , 2014, J. Netw. Comput. Appl..

[14]  Joseph Mitola,et al.  Cognitive radio: making software radios more personal , 1999, IEEE Wirel. Commun..

[15]  Albert Y. Zomaya,et al.  Big Data and Computational Intelligence in Networking , 2017 .

[16]  Bhaskar Krishnamachari,et al.  Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks , 2018, IEEE Transactions on Cognitive Communications and Networking.

[17]  Soung Chang Liew,et al.  Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks , 2017, 2018 IEEE International Conference on Communications (ICC).

[18]  Cheng-Xiang Wang,et al.  Standard Condition Number of Hessian Matrix for Neural Networks , 2019, ICC 2019 - 2019 IEEE International Conference on Communications (ICC).

[19]  Kobi Cohen,et al.  Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access , 2017, IEEE Transactions on Wireless Communications.

[20]  Senem Velipasalar,et al.  A Deep Actor-Critic Reinforcement Learning Framework for Dynamic Multichannel Access , 2019, IEEE Transactions on Cognitive Communications and Networking.

[21]  Bo Gao,et al.  Channel Aggregation in Cognitive Radio Networks with Practical Considerations , 2011, 2011 IEEE International Conference on Communications (ICC).

[22]  Tao Jiang,et al.  Deep Reinforcement Learning for Mobile Edge Caching: Review, New Features, and Open Issues , 2018, IEEE Network.

[23]  J.D. Poston,et al.  Discontiguous OFDM considerations for dynamic spectrum access in idle TV channels , 2005, First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks, 2005. DySPAN 2005..

[24]  Rong Chen,et al.  A Deep Reinforcement Learning-Based Framework for Dynamic Resource Allocation in Multibeam Satellite Systems , 2018, IEEE Communications Letters.

[25]  Zhenyu Liao,et al.  A Random Matrix Approach to Neural Networks , 2017, ArXiv.

[26]  Wei Wang,et al.  Spectrum Aggregation: Overview and Challenges , 2010, Netw. Protoc. Algorithms.

[27]  Geoffrey Ye Li,et al.  Cognitive radio networking and communications: an overview , 2011, IEEE Transactions on Vehicular Technology.

[28]  Mustafa Cenk Gursoy,et al.  ACTOR-CRITIC DEEP REINFORCEMENT LEARNING FOR DYNAMIC MULTICHANNEL ACCESS , 2018, 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[29]  Matthias Hein,et al.  The loss surface and expressivity of deep convolutional neural networks , 2017, ICLR.

[30]  Yuan Li,et al.  Learning how to Active Learn: A Deep Reinforcement Learning Approach , 2017, EMNLP.

[31]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[33]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[34]  Jeffrey Pennington,et al.  Geometry of Neural Network Loss Surfaces via Random Matrix Theory , 2017, ICML.

[35]  Haibo He,et al.  Distributive Dynamic Spectrum Access Through Deep Reinforcement Learning: A Reservoir Computing-Based Approach , 2018, IEEE Internet of Things Journal.

[36]  Wei Liang,et al.  Dynamic Spectrum Access in Cognitive Radio Networks Using Deep Reinforcement Learning and Evolutionary Game , 2018, 2018 IEEE/CIC International Conference on Communications in China (ICCC).

[37]  Shi Jin,et al.  Deep Learning for Massive MIMO CSI Feedback , 2017, IEEE Wireless Communications Letters.