A New Solution for Freeway Congestion: Cooperative Speed Limit Control Using Distributed Reinforcement Learning

This paper presents a novel variable speed limit control system under the vehicle to infrastructure environment to optimize the freeway traffic mobility and safety. The control system is a multiagent system consists of several traffic control agents. The agents work cooperatively using the proposed distributed reinforcement learning approach to maximize the freeway traffic mobility and safety benefits. The traffic mobility objective is to maintain freeway traffic density slightly under the critical point to produce the maximum traffic volume, while the traffic safety objective is to reduce the speed difference between adjacent segments. The merits of distributed reinforcement learning are its model-free nature, and it can improve its performance continually as time goes on. The control system is developed on an open source traffic simulation software. Results revealed that compared with no control cases, the proposed system can noticeably decrease the total travel time and increase the bottleneck outflow. Moreover, the speed difference between freeway segments indicating the potential rear-end collision risk is significantly reduced. We also found that there could be more than one optimal traffic equilibrium according to different control objectives, which inspire us to design more optimal strategies in the future.

[1]  Michael D Fontaine,et al.  Interaction between System Design and Operations of Variable Speed Limit Systems in Work Zones , 2010 .

[2]  Shimon Whiteson,et al.  Multiagent Reinforcement Learning for Urban Traffic Control Using Coordination Graphs , 2008, ECML/PKDD.

[3]  Mohamed Abdel-Aty,et al.  Evaluation of variable speed limits for real-time freeway safety improvement. , 2006, Accident; analysis and prevention.

[4]  A. Hegyi,et al.  Optimal Coordination of Variable Speed Limits to Suppress Shock Waves , 2002, IEEE Transactions on Intelligent Transportation Systems.

[5]  Lina Kattan,et al.  Variable speed limit: A microscopic analysis in a connected vehicle environment , 2015 .

[6]  J. Hellendoorn,et al.  Towards a practical application of model predictive control to suppress shock waves on freeways , 2007, 2007 European Control Conference (ECC).

[7]  Walid Gomaa,et al.  Multi-Agent Reinforcement Learning Control for Ramp Metering , 2014, ICSEng.

[8]  Meng Wang,et al.  Connected variable speed limits control and car-following control with vehicle-infrastructure communication to resolve stop-and-go waves , 2016, J. Intell. Transp. Syst..

[9]  Wei Wang,et al.  Reinforcement Learning-Based Variable Speed Limit Control Strategy to Reduce Traffic Congestion at Freeway Recurrent Bottlenecks , 2017, IEEE Transactions on Intelligent Transportation Systems.

[10]  Tony Z. Qiu,et al.  Cell Transmission Model-Based Variable Speed Limit Control for Freeways , 2012 .

[11]  Bart De Schutter,et al.  Multi-agent Reinforcement Learning: An Overview , 2010 .

[12]  H. JoséAntonioMartín,et al.  Robust high performance reinforcement learning through weighted k-nearest neighbors , 2011, Neurocomputing.

[13]  Michael L. Littman,et al.  Friend-or-Foe Q-learning in General-Sum Games , 2001, ICML.

[14]  Baher Abdulhai,et al.  Multiagent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC): Methodology and Large-Scale Application on Downtown Toronto , 2013, IEEE Transactions on Intelligent Transportation Systems.

[15]  Andreas Tapani,et al.  Impacts of a Cooperative Variable Speed Limit System , 2012 .

[16]  Markos Papageorgiou,et al.  Optimal mainstream traffic flow control of large scale motorway networks , 2008 .

[17]  Baher Abdulhai,et al.  Self-Learning Adaptive Ramp Metering , 2013 .

[18]  Ruben Glatt,et al.  MOO-MDP: An Object-Oriented Representation for Cooperative Multiagent Reinforcement Learning , 2019, IEEE Transactions on Cybernetics.

[19]  Meng Wang,et al.  Rolling horizon control framework for driver assistance systems. Part II: Cooperative sensing and cooperative control , 2014 .

[20]  Wei Wang,et al.  Development of a Control Strategy of Variable Speed Limits to Reduce Rear-End Collision Risks Near Freeway Recurrent Bottlenecks , 2014, IEEE Transactions on Intelligent Transportation Systems.

[21]  Martin Lauer,et al.  An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems , 2000, ICML.

[22]  Praveen Edara,et al.  Evaluation of variable advisory speed limits in congested work zones , 2017 .

[23]  Bart De Schutter,et al.  Variable speed limits for green mobility , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[24]  J. Piao,et al.  Safety Impacts of Variable Speed Limits - A Simulation Study , 2008, 2008 11th International IEEE Conference on Intelligent Transportation Systems.

[25]  Baher Abdulhai,et al.  Application of reinforcement learning with continuous state space to ramp metering in real-world conditions , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[26]  Markos Papageorgiou,et al.  Freeway ramp metering: an overview , 2002, IEEE Trans. Intell. Transp. Syst..

[27]  Lina Kattan,et al.  Variable speed limit: an overview , 2015 .

[28]  Satish V. Ukkusuri,et al.  Accounting for dynamic speed limit control in a stochastic traffic environment: a reinforcement learning approach , 2014 .