Hierarchical Deep Reinforcement Learning for Backscattering Data Collection With Multiple UAVs

The emerging backscatter communication technology is recognized as a promising solution to the battery problem of Internet of Things (IoT) devices. For example, the wireless sensor network with backscatter communication technology can monitor the environment in remote areas without battery maintenance or replacement. Unfortunately, the transmission range of backscatter communication is limited. To tackle this challenge, we propose a multi-UAV-aided data collection scenario where the unmanned aerial vehicle (UAV) can fly close to the backscatter sensor node (BSN) to activate it and then collects the data. We aim to minimize the total flight time of the rechargeable UAVs when the collection mission is finished. During the data collection process, the UAVs can return to the charging station to recharge itself when the energy of UAV is not sufficient to complete the mission. To reduce the complexity of the task, we first use the Gaussian mixture model clustering method to divide the BSNs into multiple clusters. Then we consider the deterministic boundary and ambiguous boundary for the UAV flying regions, respectively. For the deterministic boundary scenario, we propose a single-agent deep option learning (SADOL) algorithm, where each UAV cannot fly beyond the deterministic boundary. For the ambiguous boundary scenario, we propose a multiagent deep option learning (MADOL) algorithm to enable the UAVs to cooperatively learn the ambiguous BSNs assignment. In the simulation, we compare the proposed algorithms with multiagent deep deterministic policy gradient (MADDPG), deep deterministic policy gradient (DDPG), and deep Q-network (DQN) algorithms, which proves the proposed algorithms can achieve better performance.