Mixed-Autonomy Traffic Control with Proximal Policy Optimization

This work studies mixed-autonomy traffic optimization at a network level with Deep Reinforcement Learning (DRL). In mixed-autonomy traffic, a mixture of connected autonomous vehicles (CAVs) and human driving vehicles is present on the roads at the same time. We hypothesize that controlling distributed CAVs at a network level can outperform the individually controlled CAVs. Our goal is to improve traffic fluidity in terms of the vehicle's average velocity and collision avoidance. We propose three distributed learning control policies for CAVs in mixed-autonomy traffic using Proximal Policy Optimization (PPO), a policy gradient DRL method. We conduct the experiments with different traffic settings and CAV penetration rates on the Flow framework, a new open-source microscopic traffic simulator. The experiments show that network-level RL policies for controlling CAVs outperform the individual-level RL policies in terms of the total rewards and the average velocity.

[1]  Fei-Yue Wang,et al.  An Efficient Deep Reinforcement Learning Model for Urban Traffic Control , 2018, ArXiv.

[2]  Karl Henrik Johansson,et al.  String Stability and a Delay-Based Spacing Policy for Vehicle Platoons Subject to Disturbances , 2017, IEEE Transactions on Automatic Control.

[3]  Lena Mashayekhy,et al.  Intersection Management for Connected Autonomous Vehicles: A Game Theoretic Framework , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[4]  Paparao Palacharla,et al.  Cooperative autonomous driving for traffic congestion avoidance through vehicle-to-vehicle communications , 2017, 2017 IEEE Vehicular Networking Conference (VNC).

[5]  Chadi Assi,et al.  Deep reinforcement learning-based scheduling for roadside communication networks , 2017, 2017 15th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt).

[6]  Thorsten Schmidt-Dumont,et al.  Decentralised reinforcement learning for ramp metering and variable speed limits on highways , 2017 .

[7]  Li Li,et al.  Traffic signal timing via deep reinforcement learning , 2016, IEEE/CAA Journal of Automatica Sinica.

[8]  Alexandre M. Bayen,et al.  Dissipating stop-and-go waves in closed and open networks via deep reinforcement learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[9]  Wei Wang,et al.  Reinforcement Learning-Based Variable Speed Limit Control Strategy to Reduce Traffic Congestion at Freeway Recurrent Bottlenecks , 2017, IEEE Transactions on Intelligent Transportation Systems.

[10]  Mohammad Khanjary,et al.  Using game theory to optimize traffic light of an intersection , 2013, 2013 IEEE 14th International Symposium on Computational Intelligence and Informatics (CINTI).

[11]  Hesham A. Rakha,et al.  An Intersection Game-Theory-Based Traffic Control Algorithm in a Connected Vehicle Environment , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[12]  Alexandre M. Bayen,et al.  Benchmarks for reinforcement learning in mixed-autonomy traffic , 2018, CoRL.

[13]  Maria Laura Delle Monache,et al.  Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments , 2017, ArXiv.

[14]  Alexandre M. Bayen,et al.  Lagrangian Control through Deep-RL: Applications to Bottleneck Decongestion , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[15]  Ion Stoica,et al.  Ray RLLib: A Composable and Scalable Reinforcement Learning Library , 2017, NIPS 2017.

[16]  Y. Sugiyama,et al.  Traffic jams without bottlenecks—experimental evidence for the physical mechanism of the formation of a jam , 2008 .

[17]  Sergey Levine,et al.  End-to-End Training of Deep Visuomotor Policies , 2015, J. Mach. Learn. Res..

[18]  Helbing,et al.  Congested traffic states in empirical observations and microscopic simulations , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[19]  Deepeka Garg,et al.  Deep Reinforcement Learning for Autonomous Traffic Light Control , 2018, 2018 3rd IEEE International Conference on Intelligent Transportation Engineering (ICITE).

[20]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[21]  Sem C. Borst,et al.  Deep Reinforcement Learning for Intelligent Transportation Systems , 2018, ArXiv.

[22]  Yun-Pang Flötteröd,et al.  Microscopic Traffic Simulation using SUMO , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[23]  Michael I. Jordan,et al.  Ray: A Distributed Framework for Emerging AI Applications , 2017, OSDI.

[24]  Steven E Schladover.  Review of the State of Development of Advanced Vehicle Control Systems (AVCS) , 1995 .

[25]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[26]  Alexandre M. Bayen,et al.  Flow: Architecture and Benchmarking for Reinforcement Learning in Traffic Control , 2017, ArXiv.

[27]  A. Downs.  TRAFFIC: WHY IT'S GETTING WORSE, WHAT GOVERNMENT CAN DO , 2004 .