Energy Minimization for Cellular-Connected UAV: From Optimization to Deep Reinforcement Learning

Cellular-connected unmanned aerial vehicles (UAVs) are expected to become integral components of future cellular networks. To this end, one of the important problems to address is how to support energy-efficient UAV operation while maintaining reliable connectivity between those aerial users and cellular networks. In this paper, we aim to minimize the energy consumption of cellular-connected UAV via jointly designing the mission completion time and UAV trajectory, as well as communication base station (BS) associations, while ensuring a satisfactory communication connectivity with the ground cellular network during the UAV flight. An optimization problem is formulated by taking into account the UAV’s flight energy consumption and various practical aspects of the air-ground communication models, including BS antenna pattern, interference from non-associated BSs and local environment. The formulated problem is difficult to tackle due to the lack of closed-form expressions and non-convexity nature. To this end, we first assume that the channel knowledge map (CKM) or radio map for the considered area is available, which contains rich information about the relatively stable (large-scale) channel parameters. By utilizing path discretization technique, we obtain a discretized equivalent problem and develop an efficient solution based on graph theory by employing convex optimization technique and a dynamic-weight shortest path algorithm over graph. Next, we study the more practical case that the CKM is unavailable initially. By transforming the optimization problem to a Markov decision process (MDP), we develop a deep reinforcement learning (DRL) algorithm based on multi-step learning and double Q-learning over a dueling Deep Q-Network (DQN) architecture, where the UAV acts as an agent to explore and learn its moving policy according to its local observations of the measured signal samples. Extensive simulations are carried out and the results show that our proposed designs significantly outperform baseline schemes. Furthermore, our results reveal new insights of energy-efficient UAV flight with connectivity requirements and unveil the tradeoff between UAV energy consumption and time duration along line segments.

[1]  Yong Zeng,et al.  Toward Environment-Aware 6G Communications via Channel Knowledge Map , 2021, IEEE Wireless Communications.

[2]  Rui Zhang,et al.  Aerial-Ground Interference Mitigation for Cellular-Connected UAV , 2021, IEEE Wireless Communications.

[3]  Yong Zeng,et al.  Energy-Efficient Data Uploading for Cellular-Connected UAV Systems , 2020, IEEE Transactions on Wireless Communications.

[4]  George K. Karagiannidis,et al.  Energy-Efficient Resource Allocation and Trajectory Design for UAV Relaying Systems , 2020, IEEE Transactions on Communications.

[5]  Jiandong Li,et al.  Access Points in the Air: Modeling and Optimization of Fixed-Wing UAV Network , 2020, IEEE Journal on Selected Areas in Communications.

[6]  Keping Long,et al.  Energy Efficiency Optimization for NOMA UAV Network With Imperfect CSI , 2020, IEEE Journal on Selected Areas in Communications.

[7]  Qingqing Wu,et al.  Energy Management and Trajectory Optimization for UAV-Enabled Legitimate Monitoring Systems , 2020, IEEE Transactions on Wireless Communications.

[8]  Rui Zhang,et al.  Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV With Deep Reinforcement Learning , 2020, IEEE Transactions on Wireless Communications.

[9]  Yong Zeng,et al.  Aerial–Ground Cost Tradeoff for Multi-UAV-Enabled Data Collection in Wireless Sensor Networks , 2020, IEEE Transactions on Communications.

[10]  Han Hu,et al.  Unmanned Aircraft System Aided Adaptive Video Streaming: A Joint Optimization Approach , 2020, IEEE Transactions on Multimedia.

[11]  Fumiyuki Adachi,et al.  Uplink Precoding Optimization for NOMA Cellular-Connected UAV Networks , 2020, IEEE Transactions on Communications.

[12]  Shuowen Zhang,et al.  Radio Map-Based 3D Path Planning for Cellular-Connected UAV , 2019, IEEE Transactions on Wireless Communications.

[13]  Ying-Chang Liang,et al.  Energy-Efficient UAV Backscatter Communication With Joint Trajectory Design and Resource Optimization , 2019, IEEE Transactions on Wireless Communications.

[14]  Xiangyun Zhou,et al.  Uplink NOMA for Cellular-Connected UAV: Impact of UAV Trajectories and Altitude , 2019, IEEE Transactions on Communications.

[15]  Nicola Marchetti,et al.  Mobility in the Sky: Performance and Mobility Analysis for Cellular-Connected UAVs , 2019, IEEE Transactions on Communications.

[16]  Rui Zhang,et al.  Cooperative Downlink Interference Transmission and Cancellation for Cellular-Connected UAV: A Divide-and-Conquer Approach , 2019, IEEE Transactions on Communications.

[17]  Zhu Han,et al.  Completion Time Minimization With Path Planning for Fixed-Wing UAV Communications , 2019, IEEE Transactions on Wireless Communications.

[18]  Xiaoli Xu,et al.  Cellular-Connected UAV: Performance Analysis with 3D Antenna Modelling , 2019, 2019 IEEE International Conference on Communications Workshops (ICC Workshops).

[19]  Qingqing Wu,et al.  Accessing From the Sky: A Tutorial on UAV Communications for 5G and Beyond , 2019, Proceedings of the IEEE.

[20]  Rui Zhang,et al.  Uplink Cooperative NOMA for Cellular-Connected UAV , 2018, IEEE Journal of Selected Topics in Signal Processing.

[21]  Rui Zhang,et al.  Cellular-Connected UAV: Uplink Association, Power Control and Interference Coordination , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[22]  Rui Zhang,et al.  Engineering Radio Map for Wireless Resource Management , 2018, ArXiv.

[23]  Shuowen Zhang,et al.  Cellular-Enabled UAV Communication: A Connectivity-Constrained Trajectory Optimization Perspective , 2018, IEEE Transactions on Communications.

[24]  Emil Björnson,et al.  The Essential Guide to Realizing 5G-Connected UAVs with Massive MIMO , 2018, IEEE Communications Magazine.

[25]  Sofie Pollin,et al.  Cellular Connectivity for UAVs: Network Modeling, Performance Analysis, and Design Guidelines , 2018, IEEE Transactions on Wireless Communications.

[26]  Walid Saad,et al.  Echo-Liquid State Deep Learning for 360° Content Transmission and Caching in Wireless VR Networks With Cellular-Connected UAVs , 2018, IEEE Transactions on Communications.

[27]  Rui Zhang,et al.  Cellular-Connected UAV: Potential, Challenges, and Promising Technologies , 2018, IEEE Wireless Communications.

[28]  Jie Xu,et al.  Energy Minimization for Wireless Communication With Rotary-Wing UAV , 2018, IEEE Transactions on Wireless Communications.

[29]  Walid Saad,et al.  A Tutorial on UAVs for Wireless Networks: Applications, Challenges, and Open Problems , 2018, IEEE Communications Surveys & Tutorials.

[30]  Walid Saad,et al.  Interference Management for Cellular-Connected UAVs: A Deep Reinforcement Learning Approach , 2018, IEEE Transactions on Wireless Communications.

[31]  D. Gesbert,et al.  Efficient Local Map Search Algorithms for the Placement of Flying Relays , 2018, IEEE Transactions on Wireless Communications.

[32]  Sofie Pollin,et al.  Reshaping Cellular Networks for the Sky: Major Factors and Feasibility , 2017, 2018 IEEE International Conference on Communications (ICC).

[33]  Ryu Miura,et al.  A Wireless Relay Network Based on Unmanned Aircraft System With Rate Optimization , 2016, IEEE Transactions on Wireless Communications.

[34]  Rui Zhang,et al.  Energy-Efficient UAV Communication With Trajectory Optimization , 2016, IEEE Transactions on Wireless Communications.

[35]  Rui Zhang,et al.  Wireless communications with unmanned aerial vehicles: opportunities and challenges , 2016, IEEE Communications Magazine.

[36]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[37]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[38]  Marc G. Bellemare,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[39]  Mario Osvin Pavčević,et al.  Introduction to graph theory , 1973, The Mathematical Gazette.

[40]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[41]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[42]  Propagation data and prediction methods required for the design of terrestrial broadband radio access systems operating in a frequency range from 3 to 60 GHz , 2022 .