RIS-Assisted UAV Communications for IoT With Wireless Power Transfer Using Deep Reinforcement Learning

Many of the devices used in Internet-of-Things (IoT) applications are energy-limited, and thus supplying energy while maintaining seamless connectivity for IoT devices is of considerable importance. In this context, we propose a simultaneous wireless power transfer and information transmission scheme for IoT devices with support from reconfigurable intelligent surface (RIS)-aided unmanned aerial vehicle (UAV) communications. In particular, in a first phase, IoT devices harvest energy from the UAV through wireless power transfer; and then in a second phase, the UAV collects data from the IoT devices through information transmission. To characterise the agility of the UAV, we consider two scenarios: a hovering UAV and a mobile UAV. Aiming at maximizing the total network sum-rate, we jointly optimize the trajectory of the UAV, the energy harvesting scheduling of IoT devices, and the phaseshift matrix of the RIS. We formulate a Markov decision process and propose two deep reinforcement learning algorithms to solve the optimization problem of maximizing the total network sum-rate. Numerical results illustrate the effectiveness of the UAV’s flying path optimization and the network’s throughput of our proposed techniques compared with other benchmark schemes. Given the strict requirements of the RIS and UAV, the significant improvement in processing time and throughput performance demonstrates that our proposed scheme is well applicable for practical IoT applications.

[1]  Long D. Nguyen,et al.  Real-Time Energy Harvesting Aided Scheduling in UAV-Assisted D2D Networks Relying on Deep Reinforcement Learning , 2021, IEEE Access.

[2]  Minh-Nghia Nguyen,et al.  Non-Cooperative Energy Efficient Power Allocation Game in D2D Communication: A Multi-Agent Deep Reinforcement Learning Approach , 2019, IEEE Access.

[3]  Xiaohu You,et al.  Joint Beamforming and Trajectory Optimization for Intelligent Reflecting Surfaces-Assisted UAV Communications , 2020, IEEE Access.

[4]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[5]  Saman Atapattu,et al.  Reconfigurable Intelligent Surface assisted Two-Way Communications: Performance Analysis and Optimization , 2020, ArXiv.

[6]  H. Vincent Poor,et al.  Reconfigurable Intelligent Surface Assisted Device-to-Device Communications , 2020, IEEE Transactions on Wireless Communications.

[7]  Kezhi Wang,et al.  Joint Trajectory and Passive Beamforming Design for Intelligent Reflecting Surface-Aided UAV Communications: A Deep Reinforcement Learning Approach , 2020 .

[8]  Long D. Nguyen,et al.  Distributed Deep Deterministic Policy Gradient for Power Allocation Control in D2D-Based V2V Communications , 2019, IEEE Access.

[9]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[10]  Georges Kaddoum,et al.  URLLC Facilitated by Mobile UAV Relay and RIS: A Joint Design of Passive Beamforming, Blocklength, and UAV Positioning , 2021, IEEE Internet of Things Journal.

[11]  Dinh Thai Hoang,et al.  Wireless Powered Intelligent Reflecting Surfaces for Enhancing Wireless Communications , 2020, IEEE Transactions on Vehicular Technology.

[12]  Holger Claussen,et al.  3D UAV Trajectory and Data Collection Optimisation Via Deep Reinforcement Learning , 2021, IEEE Transactions on Communications.

[13]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[14]  Ronghong Mo,et al.  Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learning , 2020, IEEE Journal on Selected Areas in Communications.

[15]  Miaowen Wen,et al.  Reconfigurable Intelligent Surfaces With Reflection Pattern Modulation: Beamforming Design and Performance Analysis , 2020, IEEE Transactions on Wireless Communications.

[16]  Lajos Hanzo,et al.  Intelligent Reflecting Surface Aided MIMO Broadcasting for Simultaneous Wireless Information and Power Transfer , 2019, IEEE Journal on Selected Areas in Communications.

[17]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[18]  Saeed R. Khosravirad,et al.  Intelligent Reconfigurable Surface-assisted Multi-UAV Networks: Efficient Resource Allocation with Deep Reinforcement Learning , 2021 .

[19]  Qisheng Wang,et al.  Deep Reinforcement Learning Based Intelligent Reflecting Surface Optimization for MISO Communication Systems , 2020, IEEE Wireless Communications Letters.

[20]  Ayse Kortun,et al.  Real-Time Deployment and Resource Allocation for Distributed UAV Systems in Disaster Relief , 2019, 2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[21]  Mohamed-Slim Alouini,et al.  Wireless Communications Through Reconfigurable Intelligent Surfaces , 2019, IEEE Access.

[22]  Ying-Chang Liang,et al.  Reconfigurable Intelligent Surface Assisted UAV Communication: Joint Trajectory Design and Passive Beamforming , 2022 .

[23]  Trung Quang Duong,et al.  An Introduction of Real-time Embedded Optimisation Programming for UAV Systems under Disaster Communication , 2018, EAI Endorsed Trans. Ind. Networks Intell. Syst..

[24]  Sergey Levine,et al.  High-Dimensional Continuous Control Using Generalized Advantage Estimation , 2015, ICLR.

[25]  Xiaojun Yuan,et al.  Reconfigurable Intelligent Surface Aided Constant-Envelope Wireless Power Transfer , 2020, GLOBECOM 2020 - 2020 IEEE Global Communications Conference.

[26]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[27]  Hoang Duong Tuan,et al.  Joint Design of Reconfigurable Intelligent Surfaces and Transmit Beamforming Under Proper and Improper Gaussian Signaling , 2020, IEEE Journal on Selected Areas in Communications.

[28]  Jun Zhao,et al.  Deep Reinforcement Learning-Based Intelligent Reflecting Surface for Secure Wireless Communications , 2020, IEEE Transactions on Wireless Communications.

[29]  Qi Zhang,et al.  Joint Beamforming Design in Multi-Cluster MISO NOMA Reconfigurable Intelligent Surface-Aided Downlink Communication Networks , 2021, IEEE Transactions on Communications.