Q-LBR: Q-Learning Based Load Balancing Routing for UAV-Assisted VANET

Although various unmanned aerial vehicle (UAV)-assisted routing protocols have been proposed for vehicular ad hoc networks, few studies have investigated load balancing algorithms to accommodate future traffic growth and deal with complex dynamic network environments simultaneously. In particular, owing to the extended coverage and clear line-of-sight relay link on a UAV relay node (URN), the possibility of a bottleneck link is high. To prevent problems caused by traffic congestion, we propose Q-learning based load balancing routing (Q-LBR) through a combination of three key techniques, namely, a low-overhead technique for estimating the network load through the queue status obtained from each ground vehicular node by the URN, a load balancing scheme based on Q-learning and a reward control function for rapid convergence of Q-learning. Through diverse simulations, we demonstrate that Q-LBR improves the packet delivery ratio, network utilization and latency by more than 8, 28 and 30%, respectively, compared to the existing protocol.

[1]  Haipeng Yao,et al.  Multi-UAV-Enabled Load-Balance Mobile-Edge Computing for IoT Networks , 2020, IEEE Internet of Things Journal.

[2]  Luca Delgrossi,et al.  IEEE 802.11p: Towards an International Standard for Wireless Access in Vehicular Environments , 2008, VTC Spring 2008 - IEEE Vehicular Technology Conference.

[3]  Abbas Jamalipour,et al.  Modeling air-to-ground path loss for low altitude platforms in urban environments , 2014, 2014 IEEE Global Communications Conference.

[4]  Prathyush P. Menon,et al.  UAV Aided Dynamic Routing of Resources in a Flood Scenario , 2019, 2019 International Conference on Unmanned Aircraft Systems (ICUAS).

[5]  Xin Yuan,et al.  A Machine Learning Approach of Load Balance Routing to Support Next-Generation Wireless Networks , 2019, 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC).

[6]  Zoubir Mammeri,et al.  Reinforcement Learning Based Routing in Networks: Review and Classification of Approaches , 2019, IEEE Access.

[7]  H. T. Kung,et al.  Maximizing Throughput of UAV-Relaying Networks with the Load-Carry-and-Deliver Paradigm , 2007, 2007 IEEE Wireless Communications and Networking Conference.

[8]  Hamid Jafarkhani,et al.  UAV-aided cross-layer routing for MANETs , 2012, 2012 IEEE Wireless Communications and Networking Conference (WCNC).

[9]  Mohamed Ayoub Messous,et al.  Efficient Data Processing in Software-Defined UAV-Assisted Vehicular Networks: A Sequential Game Approach , 2018, Wirel. Pers. Commun..

[10]  Mohammad Hossein Yaghmaee Moghaddam,et al.  Design of energy-aware QoS routing protocol in wireless sensor networks using reinforcement learning , 2014, 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering (CCECE).

[11]  Fen Zhou,et al.  Intelligent UAV-assisted routing protocol for urban VANETs , 2017, Comput. Commun..

[12]  Xiang Cheng,et al.  UAV-Assisted Data Dissemination Scheduling in VANETs , 2018, 2018 IEEE International Conference on Communications (ICC).

[13]  Walid Saad,et al.  Efficient Deployment of Multiple Unmanned Aerial Vehicles for Optimal Wireless Coverage , 2016, IEEE Communications Letters.

[14]  Pascal Lorenz,et al.  U2RV: UAV‐assisted reactive routing protocol for VANETs , 2019, Int. J. Commun. Syst..

[15]  Gang Wu,et al.  Joint Resource Allocation and Trajectory Control for UAV-Enabled Vehicular Communications , 2019, IEEE Access.

[16]  Jong Wook Kim,et al.  Q-Learning Algorithms: A Comprehensive Classification and Applications , 2019, IEEE Access.

[17]  Sang-Jo Yoo,et al.  Q-Learning-Based Fuzzy Logic for Multi-objective Routing Algorithm in Flying Ad Hoc Networks , 2020, Wirel. Pers. Commun..

[18]  Wei Wu,et al.  A novel multi-step Q-learning method to improve data efficiency for deep reinforcement learning , 2019, Knowl. Based Syst..

[19]  Sayed Jobaer,et al.  UAV-Assisted Hybrid Scheme for Urban Road Safety Based on VANETs , 2020 .

[20]  Young-Bae Ko,et al.  QGeo: Q-Learning-Based Geographic Ad Hoc Routing Protocol for Unmanned Robotic Networks , 2017, IEEE Communications Letters.

[21]  Tao Luo,et al.  A Hierarchical Routing Scheme With Load Balancing in Software Defined Vehicular Ad Hoc Networks , 2018, IEEE Access.

[22]  P. Simon.  Too Big to Ignore: The Business Case for Big Data , 2013 .

[23]  Michael L. Littman,et al.  Reinforcement learning improves behaviour from evaluative feedback , 2015, Nature.

[24]  Krishna M. Sivalingam,et al.  Reinforcement Learning Based Geographic Routing Protocol for UWB Wireless Sensor Network , 2007, IEEE GLOBECOM 2007 - IEEE Global Telecommunications Conference.