Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL), a powerful datadriven control method, has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with humandriven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic network (MA2C) is developed with 1 ar X iv :2 11 1. 06 31 8v 1 [ cs .L G ] 1 1 N ov 2 02 1 Springer Nature 2021 LTEX template 2 Autonomous Intelligent Systems a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is proposed to incorporate fuel efficiency, driving comfort, and safety of autonomous driving. Comprehensive experimental results, conducted under three different traffic densities and various levels of human driver aggressiveness, show that our proposed MARL framework consistently outperforms several stateof-the-art benchmarks in terms of efficiency, safety and driver comfort.

[1]  Praveen Palanisamy,et al.  Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning , 2019, 2020 International Joint Conference on Neural Networks (IJCNN).

[2]  Tingting Li,et al.  A Cooperative Lane Change Model for Connected and Automated Vehicles , 2020, IEEE Access.

[3]  Pascal Poupart,et al.  Partially Observable Markov Decision Processes , 2010, Encyclopedia of Machine Learning.

[4]  Helbing,et al.  Congested traffic states in empirical observations and microscopic simulations , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[5]  Juan Rojas,et al.  Towards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning , 2021, IEEE Robotics and Automation Letters.

[6]  Yujie Li,et al.  A Cooperative Control Framework for CAV Lane Change in a Mixed Traffic Environment , 2020, ArXiv.

[7]  Ching-Yao Chan,et al.  Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[8]  Elman Mansimov,et al.  Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation , 2017, NIPS.

[9]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[10]  Yujie Li,et al.  A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network , 2020, ArXiv.

[11]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[12]  Yuankai Wu,et al.  Efficient Motion Planning for Automated Lane Change based on Imitation Learning and Mixed-Integer Optimization , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[13]  A. B. Rad,et al.  Lane Change Algorithm for Autonomous Vehicles via Virtual Curvature Method , 2009 .

[14]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[15]  K. Madhava Krishna,et al.  Parameter Sharing Reinforcement Learning Architecture for Multi Agent Driving Behaviors , 2018, ArXiv.

[16]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[17]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[18]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[19]  Tianshu Chu,et al.  Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control , 2019, IEEE Transactions on Intelligent Transportation Systems.

[20]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[21]  Yue Wang,et al.  Autonomous Driving using Safe Reinforcement Learning by Incorporating a Regret-based Human Lane-Changing Decision Model , 2019, 2020 American Control Conference (ACC).

[22]  Carl-Johan Hoel,et al.  Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[23]  Jonas Fredriksson,et al.  Lane Change Maneuvers for Automated Vehicles , 2017, IEEE Transactions on Intelligent Transportation Systems.

[24]  Samuel Labi,et al.  Graph neural network and reinforcement learning for multi‐agent cooperative control of connected autonomous vehicles , 2021, Comput. Aided Civ. Infrastructure Eng..

[25]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[26]  Kevin Heaslip,et al.  Minimizing the Disruption of Traffic Flow of Automated Vehicles During Lane Changes , 2015, IEEE Transactions on Intelligent Transportation Systems.

[27]  Zhiheng Li,et al.  Harmonious Lane Changing via Deep Reinforcement Learning , 2021, IEEE Transactions on Intelligent Transportation Systems.

[28]  Sandeep Chinchali,et al.  Multi-agent Reinforcement Learning for Networked System Control , 2020, ICLR.

[29]  Jonas Fredriksson,et al.  If, When, and How to Perform Lane Change Maneuvers on Highways , 2016, IEEE Intelligent Transportation Systems Magazine.

[30]  Dirk Helbing,et al.  Connectivity Statistics of Store-and-Forward Intervehicle Communication , 2010, IEEE Transactions on Intelligent Transportation Systems.

[31]  Longsheng Jiang,et al.  Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic , 2021, ArXiv.

[32]  Yujie Li,et al.  Leveraging the Capabilities of Connected and Autonomous Vehicles and Multi-Agent Reinforcement Learning to Mitigate Highway Bottleneck Congestion , 2020, ArXiv.

[33]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[34]  John M. Dolan,et al.  Attention-based Hierarchical Deep Reinforcement Learning for Lane Change Behaviors in Autonomous Driving , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[35]  Zhe Xu,et al.  Efficient Large-Scale Fleet Management via Multi-Agent Deep Reinforcement Learning , 2018, KDD.