Indirect Reinforcement Learning for Incident-responsive Ramp Control

Abstract A centralised strategy named indirect reinforcement learning ramp controller (IRLRC) has been developed in this paper to deal with ramp control problems for the congested traffic caused by incidents. IRLRC is developed on the basis of Dyna-Q architecture, under which a modified asymmetric cell transmission model (ACTM) and the standard Q-learning algorithm are combined together. The simulation-based test shows that compared with the no controlled situation, IRLRC can reduce the total travel time up to 24%, which outperforms the direct reinforcement learning (DRL) method with a reduction of 18% after the same number of iterations.

[1]  Baher Abdulhai,et al.  Machine learning for multi-jurisdictional optimal traffic corridor control , 2010 .

[2]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[3]  Mu-Han Wang.  Optimal ramp metering policies for nonrecurring congestion with uncertain incident duration , 1994 .

[4]  Richard S. Sutton,et al.  Dyna, an integrated architecture for learning, planning, and reacting , 1990, SGAR.

[5]  Mohammed Hadi,et al.  Modeling Reductions in Freeway Capacity due to Incidents in Microscopic Simulation Models , 2007 .

[6]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[7]  Mei-Shiang Chang,et al.  Stochastic Optimal-Control Approach to Automatic Incident-Responsive Coordinated Ramp Control , 2007, IEEE Transactions on Intelligent Transportation Systems.

[8]  Roberto Horowitz,et al.  Optimal freeway ramp metering using the asymmetric cell transmission model , 2006 .

[9]  Kaan Ozbay,et al.  INCIDENT MANAGEMENT IN INTELLIGENT TRANSPORTATION SYSTEMS , 1999 .

[10]  Bill Halkias,et al.  Freeway Incidents in the United States, United Kingdom, and Attica Tollway, Greece , 2008 .

[11]  Serge P. Hoogendoorn,et al.  Capacity Reduction at Incidents , 2008 .

[12]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[13]  Christopher G. Atkeson,et al.  A comparison of direct and model-based reinforcement learning , 1997, Proceedings of International Conference on Robotics and Automation.

[14]  H. Payne,et al.  Freeway ramp metering strategies for responding to incidents , 1977, 1977 IEEE Conference on Decision and Control including the 16th Symposium on Adaptive Processes and A Special Symposium on Fuzzy Set Theory and Applications.