Green Deep Reinforcement Learning for Radio Resource Management: Architecture, Algorithm Compression, and Challenges

AI heralds a step-change in the performance and capability of wireless networks and other critical infrastructures. However, it may also cause irreversible environmental damage due to their high energy consumption. Here, we address this challenge in the context of 5G and beyond, where there is a complexity explosion in radio resource management (RRM). On the one hand, deep reinforcement learning (DRL) provides a powerful tool for scalable optimization for high dimensional RRM problems in a dynamic environment. On the other hand, DRL algorithms consume a high amount of energy over time and risk compromising progress made in green radio research. This paper reviews and analyzes how to achieve green DRL for RRM via both architecture and algorithm innovations. Architecturally, a cloud based training and distributed decision-making DRL scheme is proposed, where RRM entities can make lightweight deep local decisions whilst assisted by on-cloud training and updating. On the algorithm level, compression approaches are introduced for both deep neural networks and the underlying Markov Decision Processes, enabling accurate low-dimensional representations of challenges. To scale learning across geographic areas, a spatial transfer learning scheme is proposed to further promote the learning efficiency of distributed DRL entities by exploiting the traffic demand correlations. Together, our proposed architecture and algorithms provide a vision for green and on-demand DRL capability.

[1]  Shuguang Cui,et al.  Handover Control in Wireless Systems via Asynchronous Multiuser Deep Reinforcement Learning , 2018, IEEE Internet of Things Journal.

[2]  Yiran Chen,et al.  Learning Structured Sparsity in Deep Neural Networks , 2016, NIPS.

[3]  Feng Wu,et al.  Learning Deterministic Policy with Target for Power Control in Wireless Networks , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[4]  Victor C. M. Leung,et al.  Power Control Based on Deep Reinforcement Learning for Spectrum Sharing , 2020, IEEE Transactions on Wireless Communications.

[5]  Soung Chang Liew,et al.  Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks , 2017, 2018 IEEE International Conference on Communications (ICC).

[6]  George K. Karagiannidis,et al.  Artificial Intelligence-Based Resource Allocation in Ultradense Networks: Applying Event-Triggered Q-Learning Algorithms , 2019, IEEE Vehicular Technology Magazine.

[7]  Jing Wang,et al.  A deep reinforcement learning based framework for power-efficient resource allocation in cloud RANs , 2017, 2017 IEEE International Conference on Communications (ICC).

[8]  Michael L. Littman,et al.  Near Optimal Behavior via Approximate State Abstraction , 2016, ICML.

[9]  Kobi Cohen,et al.  Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access , 2017, IEEE Transactions on Wireless Communications.

[10]  Xianfu Chen,et al.  Deep Reinforcement Learning for Resource Management in Network Slicing , 2018, IEEE Access.

[11]  Arumugam Nallanathan,et al.  Reinforcement Learning for Real-Time Optimization in NB-IoT Networks , 2019, IEEE Journal on Selected Areas in Communications.

[12]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[13]  Ying-Chang Liang,et al.  Applications of Deep Reinforcement Learning in Communications and Networking: A Survey , 2018, IEEE Communications Surveys & Tutorials.

[14]  Eryk Dutkiewicz,et al.  Optimal and Fast Real-Time Resource Slicing With Deep Dueling Neural Networks , 2019, IEEE Journal on Selected Areas in Communications.

[15]  Vivienne Sze,et al.  Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Bhaskar Krishnamachari,et al.  Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks , 2018, IEEE Transactions on Cognitive Communications and Networking.

[17]  Zhi Chen,et al.  Intelligent Power Control for Spectrum Sharing in Cognitive Radios: A Deep Reinforcement Learning Approach , 2017, IEEE Access.