An Autonomous Spectrum Management Scheme for Unmanned Aerial Vehicle Networks in Disaster Relief Operations

This paper studies the problem of spectrum shortage in an unmanned aerial vehicle (UAV) network during critical missions such as wildfire monitoring, search and rescue, and disaster monitoring. Such applications involve a high demand for high-throughput data transmissions such as real-time video-, image-, and voice- streaming where the assigned spectrum to the UAV network may not be adequate to provide the desired Quality of Service (QoS). In these scenarios, the aerial network can borrow an additional spectrum from the available terrestrial networks in trade of a relaying service for them. We propose a spectrum sharing model in which the UAVs are grouped into two classes of relaying UAVs that service the spectrum owner and the sensing UAVs that perform the disaster relief mission using the obtained spectrum. The operation of the UAV network is managed by a hierarchical mechanism in which a central controller assigns the tasks of the UAVs based on their resources and determine their operation region based on the level of priority of impacted areas and then the UAVs autonomously fine-tune their position using a model-free reinforcement learning algorithm to maximize the individual throughput and prolong their lifetime. We analyze the performance and the convergence for the proposed method analytically and with extensive simulations in different scenarios.

[1]  Gregory W. Wornell,et al.  Cooperative diversity in wireless networks: Efficient protocols and outage behavior , 2004, IEEE Transactions on Information Theory.

[2]  Sinem Coleri Ergen,et al.  Channel estimation techniques based on pilot arrangement in OFDM systems , 2002, IEEE Trans. Broadcast..

[3]  Sergey Loyka,et al.  Amplify-and-forward versus decode-and-forward relaying: which is better? , 2012 .

[4]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[5]  Mohammad Emtiyaz Khan,et al.  UAVs using Bayesian Optimization to Locate WiFi Devices , 2015, ArXiv.

[6]  Abolfazl Razi,et al.  Maximizing Energy Efficiency of Cognitive Wireless Sensor Networks With Constrained Age of Information , 2017, IEEE Transactions on Cognitive Communications and Networking.

[7]  Bernhard Rinner,et al.  Drone networks: Communications, coordination, and sensing , 2018, Ad Hoc Networks.

[8]  Yousef Darmani,et al.  Enhanced multi-route ad hoc on-demand distance vector routing , 2015, 2015 23rd Iranian Conference on Electrical Engineering.

[9]  Haitao Zhao,et al.  Deployment Algorithms of Flying Base Stations: 5G and Beyond With UAVs , 2019, IEEE Internet of Things Journal.

[10]  Peter Corcoran,et al.  Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning , 2017, ArXiv.

[11]  Abolfazl Razi,et al.  A Solution for Dynamic Spectrum Management in Mission-Critical UAV Networks , 2019, ArXiv.

[12]  Nasser Mozayani,et al.  Learning to predict where to look in interactive environments using deep recurrent q-learning , 2016, ArXiv.

[13]  R.C. Palat,et al.  Cooperative relaying for ad-hoc ground networks using swarm UAVs , 2005, MILCOM 2005 - 2005 IEEE Military Communications Conference.

[14]  Michael Schukat,et al.  Applying Q ( λ )-learning in Deep Reinforcement Learning to Play Atari Games , 2017 .

[15]  Walid Saad,et al.  Unmanned Aerial Vehicle With Underlaid Device-to-Device Communications: Performance and Tradeoffs , 2015, IEEE Transactions on Wireless Communications.

[16]  Branimir R. Vojcic,et al.  Performance of Amplify-and-Forward and Decode-and-Forward Relaying in Rayleigh Fading with Turbo Codes , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[17]  Abolfazl Razi,et al.  Distributed Cooperative Spectrum Sharing in UAV Networks Using Multi-Agent Reinforcement Learning , 2018, 2019 16th IEEE Annual Consumer Communications & Networking Conference (CCNC).

[18]  Daniel Gutiérrez-Reina,et al.  Multi-subpopulation evolutionary algorithms for coverage deployment of UAV-networks , 2018, Ad Hoc Networks.

[19]  Vuk Marojevic,et al.  Waveform and spectrum management for unmanned aerial systems beyond 2025 , 2017, 2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC).

[20]  Mark McHenry,et al.  Dynamic Spectrum Access radio performance for UAS ISR missions , 2010, 2010 - MILCOM 2010 MILITARY COMMUNICATIONS CONFERENCE.

[21]  Fatemeh Afghah,et al.  Leader-follower based coalition formation in large-scale UAV networks, a quantum evolutionary approach , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[22]  Alireza Shamsoshoara Ring Oscillator and its application as Physical Unclonable Function (PUF) for Password Management , 2019, ArXiv.

[23]  Yunfei Chen,et al.  Optimum Placement of UAV as Relays , 2018, IEEE Communications Letters.

[24]  Abdelhakim Hafid,et al.  A Bio-Inspired Solution to Cluster-Based Distributed Spectrum Allocation in High-Density Cognitive Internet of Things , 2019, IEEE Internet of Things Journal.

[25]  Dimitris A. Pados,et al.  Airborne Cognitive Networking: Design, Development, and Deployment , 2018, IEEE Access.

[26]  Dimitrios Koutsonikolas,et al.  Automating CSI Measurement with UAVs: from Problem Formulation to Energy-Optimal Solution , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[27]  Yeheskel Bar-Ness,et al.  Spectrum Leasing via Distributed Cooperation in Cognitive Radio , 2008, 2008 IEEE International Conference on Communications.

[28]  Timothy X. Brown,et al.  Policy-based radios for UAS operations , 2012, 2012 IEEE Globecom Workshops.

[29]  Fatemeh Afghah,et al.  A reputation-based stackelberg game model to enhance secrecy rate in spectrum leasing to selfish IoT devices , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[30]  Daniel Gutiérrez-Reina,et al.  A survey on unmanned aerial and aquatic vehicle multi-hop networks: Wireless communications, evaluation tools and applications , 2018, Comput. Commun..

[31]  Weifeng Su,et al.  Relay Location Optimization for Differential Amplify-and-Forward Cooperative Relaying , 2014, 2015 IEEE Global Communications Conference (GLOBECOM).

[32]  Haitao Zhao,et al.  Joint Optimization on Trajectory, Altitude, Velocity, and Link Scheduling for Minimum Mission Time in UAV-Aided Data Collection , 2020, IEEE Internet of Things Journal.

[33]  Jing Li,et al.  Amplify-forward and decode-forward: the impact of location and capacity contour , 2005, MILCOM 2005 - 2005 IEEE Military Communications Conference.

[34]  Sherali Zeadally,et al.  A Survey on Hardware-based Security Mechanisms for Internet of Things , 2019, ArXiv.

[35]  Fatemeh Afghah,et al.  Use of a quantum genetic algorithm for coalition formation in large-scale UAV networks , 2019, Ad Hoc Networks.

[36]  A. Ollero,et al.  Decentralized strategy to ensure information propagation in area monitoring missions with a team of UAVs under limited communications , 2013, 2013 International Conference on Unmanned Aircraft Systems (ICUAS).

[37]  Anthony Ephremides,et al.  A reputation-based Stackelberg game approach for spectrum sharing with cognitive cooperation , 2013, 52nd IEEE Conference on Decision and Control.

[38]  Song Jin,et al.  Optimal Node Placement and Resource Allocation for UAV Relaying Network , 2018, IEEE Communications Letters.

[39]  Evsen Yanmaz,et al.  Survey on Unmanned Aerial Vehicle Networks for Civil Applications: A Communications Viewpoint , 2016, IEEE Communications Surveys & Tutorials.

[40]  Aziz Mohaisen,et al.  When Smart TV Meets CRN: Privacy-Preserving Fine-Grained Spectrum Access , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[41]  Aria Nosratinia,et al.  Cooperative communication in wireless networks , 2004, IEEE Communications Magazine.

[42]  Pini Gurfil,et al.  Distributed Decision and Control for Cooperative UAVs Using Ad Hoc Communication , 2008, IEEE Transactions on Control Systems Technology.

[43]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[44]  Michael I. Jordan,et al.  MASSACHUSETTS INSTITUTE OF TECHNOLOGY ARTIFICIAL INTELLIGENCE LABORATORY and CENTER FOR BIOLOGICAL AND COMPUTATIONAL LEARNING DEPARTMENT OF BRAIN AND COGNITIVE SCIENCES , 1996 .

[45]  Xingqin Lin,et al.  An Overview of 3GPP Release-15 Study on Enhanced LTE Support for Connected Drones , 2018, IEEE Communications Standards Magazine.

[46]  Abolfazl Razi,et al.  An Online Learning Method to Maximize Energy Efficiency of Cognitive Sensor Networks , 2018, IEEE Communications Letters.

[47]  Haitao Zhao,et al.  Deployment Algorithms for UAV Airborne Networks Toward On-Demand Coverage , 2018, IEEE Journal on Selected Areas in Communications.

[48]  Andres Kwasinski,et al.  Deep Reinforcement Learning Approach to QoE-Driven Resource Allocation for Spectrum Underlay in Cognitive Radio Networks , 2018, 2018 IEEE International Conference on Communications Workshops (ICC Workshops).

[49]  Wei Zhang,et al.  Spectrum Sharing for Drone Networks , 2017, IEEE Journal on Selected Areas in Communications.

[50]  Qinfang Sun,et al.  Estimation of continuous flat fading MIMO channels , 2002, 2002 IEEE Wireless Communications and Networking Conference Record. WCNC 2002 (Cat. No.02TH8609).

[51]  Zhu Han,et al.  Spectrum Sharing Planning for Full-Duplex UAV Relaying Systems With Underlaid D2D Communications , 2018, IEEE Journal on Selected Areas in Communications.

[52]  Rui Zhang,et al.  Spectrum Sharing and Cyclical Multiple Access in UAV-Aided Cellular Offloading , 2017, GLOBECOM 2017 - 2017 IEEE Global Communications Conference.

[53]  Andres Kwasinski,et al.  QoE-Driven Integrated Heterogeneous Traffic Resource Allocation Based on Cooperative Learning for 5G Cognitive Radio Networks , 2018, 2018 IEEE 5G World Forum (5GWF).

[54]  Fatemeh Afghah,et al.  Spectrum sharing in cooperative cognitive radio networks: A matching game framework , 2015, 2015 49th Annual Conference on Information Sciences and Systems (CISS).

[55]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[56]  Daniel Gutiérrez-Reina,et al.  Evolutionary deployment and local search-based movements of 0th responders in disaster scenarios , 2018, Future Gener. Comput. Syst..

[57]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[58]  Timothy X. Brown,et al.  Secure Distribution of Spectrum Policies to Cognitive Radios in Unmanned Aircraft Systems , 2010 .

[59]  Alireza Shamsoshoara,et al.  Overview of Blakley's Secret Sharing Scheme , 2019, ArXiv.

[60]  Dongfeng Yuan,et al.  Distributed Resource Management for Cognitive Ad Hoc Networks With Cooperative Relays , 2016, IEEE/ACM Transactions on Networking.