Dynamic Multi-channel Access in Wireless System with Deep Reinforcement Learning

Dynamic spectrum access (DSA) has been considered a promising technology to address the scarcity of spectrum. In this paper, We investigate the use of deep reinforcement learning (DRL) to solve the DSA problem by maximizing the expected long-term normalized reward. Specifically, we propose a deep Q-network (DQN) based algorithm to solve dynamic multichannel access problem where there are multiple orthogonal channels in the wireless system. The problem is formulated as a Partially Observable Markov Decision Process (POMDP) with partially unknown system dynamics. We train the agent by interacting with the built system environment to find the optimal access policy of the single user and combine deep neural network to reduce the computational complexity caused by large action space. Finally, we compare our DRL approach with two conventional algorithms, and the simulation results show it can achieve near-optimal performance in a dynamic system, and we also compare our DQN algorithm with Double DQN (DDQN) algorithm which can achieve better performance and address overestimating action values.

[1]  Kobi Cohen,et al.  Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access , 2017, IEEE Transactions on Wireless Communications.

[2]  R. Srikant,et al.  Distributed Learning Algorithms for Spectrum Sharing in Spatial Random Access Wireless Networks , 2015, IEEE Transactions on Automatic Control.

[3]  Brian M. Sadler,et al.  A Survey of Dynamic Spectrum Access , 2007, IEEE Signal Processing Magazine.

[4]  Shimon Whiteson,et al.  Learning to Communicate with Deep Multi-Agent Reinforcement Learning , 2016, NIPS.

[5]  Anna Scaglione,et al.  Restless multi-armed bandits under time-varying activation constraints for dynamic spectrum access , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[6]  R. Michael Buehrer,et al.  Deep Reinforcement Learning for Dynamic Spectrum Access in Wireless Networks , 2018, MILCOM 2018 - 2018 IEEE Military Communications Conference (MILCOM).

[7]  Ian F. Akyildiz,et al.  NeXt generation/dynamic spectrum access/cognitive radio wireless networks: A survey , 2006, Comput. Networks.

[8]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[9]  Bhaskar Krishnamachari,et al.  Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks , 2018, IEEE Transactions on Cognitive Communications and Networking.

[10]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[11]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.