Optimal UAV Base Station Trajectories Using Flow-Level Models for Reinforcement Learning

Cellular base stations (BS) and remote radio heads can be mounted on unmanned aerial vehicles (UAV) for flexible, traffic-aware deployment. These UAV base station networks (UAVBSN) promise an unprecendented degree of freedom that can be exploited for spectral efficiency gains as well as optimal network utilization. However, the current literature lacks realistic radio and traffic models for UAVBSN deployment planning and for performance evaluation. In this paper, we propose flow-level models (FLM) for realistically characterizing the UAVBSN performance in terms of a broad range of flow- and system-level metrics. Further, we propose a deep reinforcement learning (DRL) approach that relies on the UAVBSN FLM for learning the optimal traffic-aware UAV trajectories. For a given user traffic density and starting UAV locations, our RL approach learns the optimal UAV trajectories offline that maximizes a cumulative performance metric. We then execute the learned UAV trajectories in a discrete event simulator to evaluate online UAVBSN performance. For ${M}\pmb {=}$ 9 UAVs deployed in a simulated Downtown San Francisco model, where the UAV trajectories are defined by ${N}\pmb {=}$ 20 discrete actions, our approach achieves approximately a three-fold increase in the average user throughput compared to the initial UAV placement, while simultaneously balancing traffic loads across the BSs.