Motorway ramp-metering control with queuing consideration using Q-learning

The standard reinforcement learning algorithms have proven to be effective tools for letting an agent learn from its experiences generated by its interaction with an environment. Among others, reinforcement learning algorithms are of interest because they require no explicit model of the environment beforehand and learning happens through trial and error. This property makes them suitable for real control problems like traffic control. Especially when considering the performance of a network where for instance a local ramp-metering controller needs to consider the performance of the network, since limitations needs to be considered, like the maximum permissible queue length, reinforcement learning algorithms are of interest. Here, a local ramp-metering control problem with queuing consideration is taken up and the performance of standard Q-learning algorithm as well as a newly proposed multi-criterion reinforcement learning algorithm is investigated. The experimental analysis confirms that the proposed multi-criterion control approach has the capability to decrease the state-space size and increase the learning speed of controller while improving the quality of solution.

[1]  Shalabh Bhatnagar,et al.  Reinforcement Learning With Function Approximation for Traffic Signal Control , 2011, IEEE Transactions on Intelligent Transportation Systems.

[2]  Jan H. van Schuppen,et al.  A hierarchical model and implementation architecture for road traffic control , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[3]  Markos Papageorgiou,et al.  An integrated control approach for traffic corridors , 1995 .

[4]  Baher Abdulhai,et al.  Integrated traffic corridor control using machine learning , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[5]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[6]  Markos Papageorgiou,et al.  METANET: A MACROSCOPIC SIMULATION PROGRAM FOR MOTORWAY NETWORKS , 1990 .

[7]  Yumei Zhang,et al.  A Machine Learning Method for Dynamic Traffic Control and Guidance on Freeway Networks , 2009, 2009 International Asia Conference on Informatics in Control, Automation and Robotics.

[8]  Markos Papageorgiou,et al.  Optimal Coordinated Ramp Metering with Advanced Motorway Optimal Control , 2001 .

[9]  B. De Moor,et al.  Anticipative model predictive control for ramp metering in freeway networks , 2003, Proceedings of the 2003 American Control Conference, 2003..

[10]  Andras Hegyi,et al.  Model predictive control for integrating traffic control measures , 2004 .

[11]  Ana L. C. Bazzan,et al.  Learning in groups of traffic signals , 2010, Eng. Appl. Artif. Intell..

[12]  Bart De Schutter,et al.  Model predictive control for optimal coordination of ramp metering and variable speed limits , 2005 .

[13]  Dipti Srinivasan,et al.  REAL-TIME COORDINATED SIGNAL CONTROL USING AGENTS WITH ONLINE REINFORCEMENT LEARNING , 2003 .

[14]  Jan M. Maciejowski,et al.  Predictive control : with constraints , 2002 .

[15]  Mohsen Davarynejad,et al.  Multi-phase time series models for motorway flow forecasting , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[16]  Toru Nakamura.  WHITE PAPER, European transport policy for 2010 : time to decide , 2004 .

[17]  Andreas Hegyi,et al.  FREEWAY TRAFFIC CONTROL USING Q-LEARNING , 2010 .

[18]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[19]  Xiaofeng Ji,et al.  An Optimal Control Method for Expressways Entering Ramps Metering Based on Q-Learning , 2009, 2009 Second International Conference on Intelligent Computation Technology and Automation.

[20]  Louis Wehenkel,et al.  Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[21]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[22]  Dipti Srinivasan,et al.  Real-Time Coordinated Signal Control Through Use of Agents with Online Reinforcement Learning , 2003 .

[23]  Ana L. C. Bazzan,et al.  Opportunities for multiagent systems and multiagent reinforcement learning in traffic control , 2009, Autonomous Agents and Multi-Agent Systems.

[24]  Jeffrey O. Kephart,et al.  The Vision of Autonomic Computing , 2003, Computer.

[25]  Baher Abdulhai,et al.  Reinforcement learning for true adaptive traffic signal control , 2003 .

[26]  Markos Papageorgiou,et al.  SERIES OF NEW LOCAL RAMP METERING STRATEGIES , 2003 .