Q-Learning Applied to Soft-Kill Countermeasures For Unmanned Aerial Vehicles (UAVs)

This work presents a three-dimensional control algorithm using reinforcement learning to guide an attacking hunter drone capable of performing a global navigation satellite systems (GNSS) repeater attack on the GNSS receiver of a target invader drone. Considering the mission and movement requirements of the hunter drone, a Q-learning algorithm was developed, for which the table with the possible transitions of the states and actions is obtained by the actions that the vehicle can take considering directions and the respective consequences of each action. The learning capability of the proposed algorithm arises from the trial and error by an agent. The penalty calculation is based on the error of the invader position to the hunter's desired position of the attacked drone. The developed algorithm is tested using a software-in-the-loop (SITL) implementation, which is based on the Ardupilot platform. SITL simulations are performed in a developed testbed to emulate operational scenarios, where an unmanned aerial vehicle (UAV) is hijacked and then controlled by an attacking UAV until it reaches the final position desired by the hunter, usually a secure area where the vehicle can be captured without being destroyed. Results, including error metrics and action time, are discussed for different mission scenarios.