Fully-Echoed Q-Routing With Simulated Annealing Inference for Flying Adhoc Networks

Current networking protocols deem inefficient in accommodating the two key challenges of Unmanned Aerial Vehicle (UAV) networks, namely the network connectivity loss and energy limitations. One approach to solve these issues is using learning-based routing protocols to make close-to-optimal local decisions by the network nodes, and Q-routing is a bold example of such protocols. However, the performance of the current implementations of Q-routing algorithms is not yet satisfactory, mainly due to the lack of adaptability to continued topology changes. In this paper, we propose a full-echo Q-routing algorithm with a self-adaptive learning rate that utilizes Simulated Annealing (SA) optimization to control the exploration rate of the algorithm through the temperature decline rate, which in turn is regulated by the experienced variation rate of the Q-values. Our results show that our method adapts to the network dynamicity without the need for manual re-initialization at transition points (abrupt network topology changes). Our method exhibits a reduction in the energy consumption ranging from 7% up to 82%, as well as a 2.6 fold gain in successful packet delivery rate, compared to the state of the art Q-routing protocols.

[1]  Yuting Zhang,et al.  Predictive routing for wireless networks: Robotics-based test and evaluation platform , 2018, 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC).

[2]  Fatemeh Afghah,et al.  An Autonomous Spectrum Management Scheme for Unmanned Aerial Vehicle Networks in Disaster Relief Operations , 2019, IEEE Access.

[3]  Ekram Khan,et al.  Aerial surveillance system using UAV , 2016, 2016 Thirteenth International Conference on Wireless and Optical Communications Networks (WOCN).

[4]  Tracy Camp,et al.  A survey of mobility models for ad hoc network research , 2002, Wirel. Commun. Mob. Comput..

[5]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[6]  J. J. Garcia-Luna-Aceves,et al.  An efficient routing protocol for wireless networks , 1996, Mob. Networks Appl..

[7]  Niyati Gupta,et al.  Improved Route Selection Approaches using Q-learning framework for 2D NoCs , 2015, MES@ISCA.

[8]  Guanghua Song,et al.  QNGPSR: A Q-Network Enhanced Geographic Ad-Hoc Routing Protocol Based on GPSR , 2018, 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall).

[9]  Konstantinos Kanistras,et al.  A survey of unmanned aerial vehicles (UAVs) for traffic monitoring , 2013, 2013 International Conference on Unmanned Aircraft Systems (ICUAS).

[10]  Fatemeh Afghah,et al.  Wildfire Spread Modeling with Aerial Image Processing , 2020, 2020 IEEE 21st International Symposium on "A World of Wireless, Mobile and Multimedia Networks" (WoWMoM).

[11]  Zhu Han,et al.  Taking Drones to the Next Level: Cooperative Distributed Unmanned-Aerial-Vehicular Networks for Small and Mini Drones , 2017, IEEE Vehicular Technology Magazine.

[12]  Michael L. Littman,et al.  Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach , 1993, NIPS.

[13]  Katia Jaffrès-Runser,et al.  QMR: Q-learning based Multi-objective optimization Routing protocol for Flying Ad Hoc Networks , 2020, Comput. Commun..

[14]  Abolfazl Razi,et al.  Optimal Measurement Policy for Linear Measurement Systems With Applications to UAV Network Topology Prediction , 2020, IEEE Transactions on Vehicular Technology.

[15]  Manpreet Kaur,et al.  Poisson's probability‐based Q‐Routing techniques for message forwarding in opportunistic networks , 2018, Int. J. Commun. Syst..

[16]  Yang Liu,et al.  A new Q-learning algorithm based on the metropolis criterion , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Douglas R. Heisterkamp,et al.  Simulated Annealing Based Hierarchical Q-Routing: A Dynamic Routing Protocol , 2011, 2011 Eighth International Conference on Information Technology: New Generations.

[18]  Chenguang He,et al.  A Fuzzy Logic Reinforcement Learning-Based Routing Algorithm For Flying Ad Hoc Networks , 2020, 2020 International Conference on Computing, Networking and Communications (ICNC).

[19]  David A. Maltz,et al.  Dynamic Source Routing in Ad Hoc Wireless Networks , 1994, Mobidata.

[20]  Charles E. Perkins,et al.  Highly dynamic Destination-Sequenced Distance-Vector routing (DSDV) for mobile computers , 1994, SIGCOMM.

[21]  Tao Wang,et al.  Adaptive Communication Protocols in Flying Ad Hoc Network , 2018, IEEE Communications Magazine.

[22]  Fang Wang,et al.  Dynamic Routing Algorithm with Q-learning for Internet of things with Delayed Estimator , 2019, IOP Conference Series: Earth and Environmental Science.

[23]  C. Udriste,et al.  Maximum Entropy Distributions , 2014 .

[24]  Philippe Jacquet,et al.  Optimized Link State Routing Protocol (OLSR) , 2003, RFC.

[25]  Ouns Bouachir,et al.  A mobility model for UAV ad hoc network , 2014, 2014 International Conference on Unmanned Aircraft Systems (ICUAS).

[26]  Yuefeng Ji,et al.  Adaptive UAV-Assisted Geographic Routing With Q-Learning in VANET , 2021, IEEE Communications Letters.

[27]  Nguyen H. Tran,et al.  AFRL: Adaptive federated reinforcement learning for intelligent jamming defense in FANET , 2020, Journal of Communications and Networks.

[28]  Yuliya Shilova,et al.  Adaptive Q-Routing with random echo and route memory , 2017, 2017 20th Conference of Open Innovations Association (FRUCT).

[29]  Matthias R. Brust,et al.  A networked swarm model for UAV deployment in the assessment of forest environments , 2015, 2015 IEEE Tenth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP).

[30]  Christian Wietfeld,et al.  PARRoT: Predictive Ad-hoc Routing Fueled by Reinforcement Learning and Trajectory Knowledge , 2020, 2021 IEEE 93rd Vehicular Technology Conference (VTC2021-Spring).

[31]  Ian F. Akyildiz,et al.  Help from the Sky: Leveraging UAVs for Disaster Management , 2017, IEEE Pervasive Computing.

[32]  Sang-Jo Yoo,et al.  Q-Learning-Based Fuzzy Logic for Multi-objective Routing Algorithm in Flying Ad Hoc Networks , 2020, Wirel. Pers. Commun..

[33]  Eleni I. Vlahogianni,et al.  Unmanned Aerial Aircraft Systems for Transportation Engineering: Current Practice and Future Challenges , 2016 .

[34]  Sidi-Mohammed Senouci,et al.  Network connectivity and area coverage for UAV fleet mobility model with energy constraint , 2016, 2016 IEEE Wireless Communications and Networking Conference.

[35]  Yuliya Shilova,et al.  Full Echo Q-routing with adaptive learning rates: A reinforcement learning approach to network routing , 2016, 2016 IEEE NW Russia Young Researchers in Electrical and Electronic Engineering Conference (EIConRusNW).

[36]  Xiaohong Jiang,et al.  A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks , 2020, Frontiers of Information Technology & Electronic Engineering.

[37]  Chunxiao Jiang,et al.  Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks , 2019, IEEE Communications Surveys & Tutorials.

[38]  Abolfazl Razi,et al.  Fully Autonomous UAV-Based Action Recognition System Using Aerial Imagery , 2020, ISVC.

[39]  Claudio E. Palazzi,et al.  A Comparison of Stateless Position-based Packet Routing Algorithms for FANETs , 2018, IEEE Transactions on Mobile Computing.

[40]  Young-Bae Ko,et al.  QGeo: Q-Learning-Based Geographic Ad Hoc Routing Protocol for Unmanned Robotic Networks , 2017, IEEE Communications Letters.

[41]  Sunil Kr. Maakar,et al.  A Survey: Different Mobility Model for FANET , 2015 .

[42]  Zabih Ghassemlooy,et al.  Comparison of the Q-routing and shortest path routing algorithm , 2004 .

[43]  Risto Miikkulainen,et al.  Confidence Based Dual Reinforcement Q-Routing: An adaptive online network routing algorithm , 1999, IJCAI.

[44]  Dit-Yan Yeung,et al.  Predictive Q-Routing: A Memory-based Reinforcement Learning Approach to Adaptive Traffic Control , 1995, NIPS.

[45]  Jianmin LIU,et al.  ARdeep: Adaptive and Reliable Routing Protocol for Mobile Robotic Networks with Deep Reinforcement Learning , 2020, 2020 IEEE 45th Conference on Local Computer Networks (LCN).

[46]  Mohammad Javad Yazdanpanah,et al.  Network routing based on reinforcement learning in dynamically changing networks , 2005, 17th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'05).

[47]  Abolfazl Razi,et al.  Predictive routing for dynamic UAV networks , 2017, 2017 IEEE International Conference on Wireless for Space and Extreme Environments (WiSEE).

[48]  Steven Latré,et al.  Q2-Routing : A Qos-aware Q-Routing algorithm for Wireless Ad Hoc Networks , 2018, 2018 14th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob).

[49]  J. Broch,et al.  Dynamic source routing in ad hoc wireless networks , 1998 .

[50]  Fatemeh Afghah,et al.  On Greedy Routing in Dynamic UAV Networks , 2018, 2018 IEEE International Conference on Sensing, Communication and Networking (SECON Workshops).

[51]  Lav Gupta,et al.  Survey of Important Issues in UAV Communication Networks , 2016, IEEE Communications Surveys & Tutorials.

[52]  Brad Karp,et al.  GPSR: greedy perimeter stateless routing for wireless networks , 2000, MobiCom '00.

[53]  Doina Bein,et al.  Optimizing Border Patrol Operations Using Unmanned Aerial Vehicles , 2015, 2015 12th International Conference on Information Technology - New Generations.

[54]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[55]  Abolfazl Razi,et al.  Aerial Imagery Pile burn detection using Deep Learning: the FLAME dataset , 2020, Comput. Networks.

[56]  Agathoniki Trigoni,et al.  Supporting Search and Rescue Operations with UAVs , 2010, 2010 International Conference on Emerging Security Technologies.