A Model-Driven Deep Reinforcement Learning Heuristic Algorithm for Resource Allocation in Ultra-Dense Cellular Networks

Resource allocation in ultra dense network (UDN) is an multi-objective optimization problem since it has to consider the tradeoff among spectrum efficiency (SE), energy efficiency (EE) and fairness. The existing methods can not effectively solve this NP-hard nonconvex problem, especially in the presence of limited channel state information (CSI). In this paper, we investigate a novel model-driven deep reinforcement learning assisted resource allocation method. We first design a novel deep neural network (DNN)-based optimization framework consisting of a series of Alternating Direction Method of Multipliers (ADMM) iterative procedures, which makes the CSI as the learned weights. Then a novel channel information absent Q-learning resource allocation (CIAQ) algorithm is proposed to train the DNN-based optimization framework without massive labeling data, where the SE, the EE, and the fairness can be jointly optimized by adjusting discount factor. Our simulation results show that, the proposed CIAQ with rapid convergence speed not only well characterizes the extent of optimization objective with partial CSI, but also significantly outperforms the current random initialization method of neural network and the other existing resource allocation algorithms in term of the tradeoff among the SE, EE and fairness.

[1]  I Chih-Lin,et al.  Energy and Spectral Efficient Frequency Reuse of Ultra Dense Networks , 2016, IEEE Transactions on Wireless Communications.

[2]  Geoffrey Ye Li,et al.  Deep Reinforcement Learning Based Resource Allocation for V2V Communications , 2018, IEEE Transactions on Vehicular Technology.

[3]  Xinyu Gu,et al.  A self-organizing resource allocation strategy based on Q-learning approach in ultra-dense networks , 2016, 2016 IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC).

[4]  Nei Kato,et al.  A Deep-Learning-Based Radio Resource Assignment Technique for 5G Ultra Dense Networks , 2018, IEEE Network.

[5]  Victor C. M. Leung,et al.  Downlink Energy Efficiency of Power Allocation and Wireless Backhaul Bandwidth Allocation in Heterogeneous Small Cell Networks , 2017, IEEE Transactions on Communications.

[6]  Wen Wang,et al.  A Cluster-Based Energy-Efficient Resource Management Scheme for Ultra-Dense Networks , 2016, IEEE Access.

[7]  Lajos Hanzo,et al.  Multiple Access Design for Ultra-Dense VLC Networks: Orthogonal vs Non-Orthogonal , 2019, IEEE Transactions on Communications.

[8]  Xin Su,et al.  A QoS-Based Cross-Tier Cooperation Resource Allocation Scheme Over Ultra-Dense HetNets , 2019, IEEE Access.

[9]  Wentao Zhao,et al.  Resource Sharing Scheme for Device-to-Device Communication Underlaying Cellular Networks , 2015, IEEE Transactions on Communications.

[10]  Ignas G. Niemegeers,et al.  Fairness in Wireless Networks:Issues, Measures and Challenges , 2014, IEEE Communications Surveys & Tutorials.

[11]  Zhi Ding,et al.  Mixed Mode Transmission and Resource Allocation for D2D Communication , 2016, IEEE Transactions on Wireless Communications.

[12]  Gang Feng,et al.  Intelligent Resource Scheduling for 5G Radio Access Network Slicing , 2019, IEEE Transactions on Vehicular Technology.

[13]  Eitan Altman,et al.  User Association and Resource Allocation Optimization in LTE Cellular Networks , 2017, IEEE Transactions on Network and Service Management.

[14]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[15]  Hüseyin Arslan,et al.  Joint Optimization of Device to Device Resource and Power Allocation Based on Genetic Algorithm , 2018, IEEE Access.

[16]  Song Guo,et al.  Green Resource Allocation Based on Deep Reinforcement Learning in Content-Centric IoT , 2018, IEEE Transactions on Emerging Topics in Computing.

[17]  Won-Joo Hwang,et al.  Fairness-Aware Spectral and Energy Efficiency in Spectrum-Sharing Wireless Networks , 2017, IEEE Transactions on Vehicular Technology.

[18]  Zongben Xu,et al.  ADMM-CSNet: A Deep Learning Approach for Image Compressive Sensing , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Cheng-Xiang Wang,et al.  Spectral, Energy, and Economic Efficiency of 5G Multicell Massive MIMO Systems With Generalized Spatial Modulation , 2016, IEEE Transactions on Vehicular Technology.

[20]  Tiejun Lv,et al.  Deep Q-Learning Based Dynamic Resource Allocation for Self-Powered Ultra-Dense Networks , 2018, 2018 IEEE International Conference on Communications Workshops (ICC Workshops).

[21]  Muhammad Ali Imran,et al.  Wireless Backhaul: Performance Modeling and Impact on User Association for 5G , 2018, IEEE Transactions on Wireless Communications.

[22]  Kai-Kit Wong,et al.  Energy-Efficient Heterogeneous Cellular Networks With Spectrum Underlay and Overlay Access , 2016, IEEE Transactions on Vehicular Technology.

[23]  Zhu Han,et al.  Spectrum Allocation and Power Control in Full-Duplex Ultra-Dense Heterogeneous Networks , 2019, IEEE Transactions on Communications.

[24]  Xiang Cheng,et al.  Graph Coloring Based Resource Sharing (GCRS) Scheme for D2D Communications Underlaying Full-Duplex Cellular Networks , 2017, IEEE Transactions on Vehicular Technology.

[25]  Lajos Hanzo,et al.  Graph-Based Joint User-Centric Overlapped Clustering and Resource Allocation in Ultradense Networks , 2018, IEEE Transactions on Vehicular Technology.

[26]  John M. Cioffi,et al.  Spatial Reuse in Dense Wireless Areas: A Cross-Layer Optimization Approach via ADMM , 2015, IEEE Transactions on Wireless Communications.

[27]  Nei Kato,et al.  Routing or Computing? The Paradigm Shift Towards Intelligent Computer Network Packet Transmission Based on Deep Learning , 2017, IEEE Transactions on Computers.

[28]  Yu Zhang,et al.  Intelligent Cloud Resource Management with Deep Reinforcement Learning , 2018, IEEE Cloud Computing.

[29]  Robert Schober,et al.  Online Resource Allocation for Energy Harvesting Downlink Multiuser Systems: Precoding With Modulation, Coding Rate, and Subchannel Selection , 2015, IEEE Transactions on Wireless Communications.

[30]  Weng Chon Ao,et al.  Approximation Algorithms for Online User Association in Multi-Tier Multi-Cell Mobile Networks , 2017, IEEE/ACM Transactions on Networking.

[31]  Lingyang Song,et al.  Load Balancing for 5G Ultra-Dense Networks Using Device-to-Device Communications , 2018, IEEE Transactions on Wireless Communications.

[32]  Lajos Hanzo,et al.  Weighted Sum-Rate Maximization for the Ultra-Dense User-Centric TDD C-RAN Downlink Relying on Imperfect CSI , 2019, IEEE Transactions on Wireless Communications.

[33]  Tho Le-Ngoc,et al.  Limited-Feedback Resource Allocation in Heterogeneous Cellular Networks , 2016, IEEE Transactions on Vehicular Technology.

[34]  Fa-Long Luo,et al.  Signal processing for 5G : algorithms and implementations , 2016 .

[35]  Ahmed Abdel-Hadi,et al.  Resource allocation in cellular systems for applications with random parameters , 2015, 2016 International Conference on Computing, Networking and Communications (ICNC).

[36]  Weidong Wang,et al.  Deep Reinforcement Learning Based Dynamic Channel Allocation Algorithm in Multibeam Satellite Systems , 2018, IEEE Access.

[37]  Zongben Xu,et al.  Model-driven deep-learning , 2018 .