A Survey on Reinforcement Learning Models and Algorithms for Traffic Signal Control

Traffic congestion has become a vexing and complex issue in many urban areas. Of particular interest are the intersections where traffic bottlenecks are known to occur despite being traditionally signalized. Reinforcement learning (RL), which is an artificial intelligence approach, has been adopted in traffic signal control for monitoring and ameliorating traffic congestion. RL enables autonomous decision makers (e.g., traffic signal controllers) to observe, learn, and select the optimal action (e.g., determining the appropriate traffic phase and its timing) to manage traffic such that system performance is improved. This article reviews various RL models and algorithms applied to traffic signal control in the aspects of the representations of the RL model (i.e., state, action, and reward), performance measures, and complexity to establish a foundation for further investigation in this research field. Open issues are presented toward the end of this article to discover new research areas with the objective to spark new interest in this research field.

[1]  Abdellah El Moudni,et al.  Traffic network micro-simulation model and control algorithm based on approximate dynamic programming , 2016 .

[2]  Jianqiang Yi,et al.  A comparative study of urban traffic signal control with reinforcement learning and Adaptive Dynamic Programming , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[3]  Walid Gomaa,et al.  Multi-objective traffic light control system based on Bayesian probability interpretation , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[4]  Dongbin Zhao,et al.  Computational Intelligence in Urban Traffic Signal Control: A Survey , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[5]  A. Koopman,et al.  Simulation and optimization of traffic in a city , 2004, IEEE Intelligent Vehicles Symposium, 2004.

[6]  Yukinori Kakazu,et al.  Genetic reinforcement learning for cooperative traffic signal control , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[7]  K.T.K. Teo,et al.  Optimization of Traffic Flow within an Urban Traffic Light Intersection with Genetic Algorithm , 2010, 2010 Second International Conference on Computational Intelligence, Modelling and Simulation.

[8]  Carlos Gershenson,et al.  Modeling self-organizing traffic lights with elementary cellular automata , 2009, ArXiv.

[9]  Mohamed A. Khamis,et al.  Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework , 2014, Eng. Appl. Artif. Intell..

[10]  Kenneth Tze Kin Teo,et al.  Q-Learning Traffic Signal Optimization within Multiple Intersections Traffic Network , 2012, 2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation.

[11]  Shalabh Bhatnagar,et al.  Decentralized learning for traffic signal control , 2015, 2015 7th International Conference on Communication Systems and Networks (COMSNETS).

[12]  Dipti Srinivasan,et al.  Urban traffic signal control using reinforcement learning agents , 2010 .

[13]  R. D. Bretherton,et al.  Optimizing networks of traffic signals in real time-the SCOOT method , 1991 .

[14]  Mohamed A. Khamis,et al.  Enhanced multiagent multi-objective reinforcement learning for urban traffic light control , 2012, 2012 11th International Conference on Machine Learning and Applications.

[15]  Wang Meng,et al.  Urban Traffic Signal Learning Control Using Fuzzy Actor-Critic Methods , 2009, 2009 Fifth International Conference on Natural Computation.

[16]  Ana L. C. Bazzan,et al.  I TSUMO: an Agent-Based Simulator for ITS Applications , 2010 .

[17]  Michael Schreckenberg,et al.  A cellular automaton model for freeway traffic , 1992 .

[18]  Shiru Qu,et al.  A stochastic adaptive traffic signal control model based on fuzzy reinforcement learning , 2010, 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE).

[19]  Chin Kian Keong.  THE GLIDE SYSTEM : SINGAPORE'S URBAN TRAFFIC CONTROL SYSTEM , 1993 .

[20]  Baher Abdulhai,et al.  Multiagent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC): Methodology and Large-Scale Application on Downtown Toronto , 2013, IEEE Transactions on Intelligent Transportation Systems.

[21]  M.A. Khamis,et al.  Adaptive traffic control system based on Bayesian probability interpretation , 2012, 2012 Japan-Egypt Conference on Electronics, Communications and Computers.

[22]  Abbas Khosravi,et al.  Q-learning method for controlling traffic signal phase time in a single intersection , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[23]  Xiaoliang Ma,et al.  Adaptive Group-Based Signal Control Using Reinforcement Learning with Eligibility Traces , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[24]  Naser Pariz,et al.  A Novel Fuzzy Model and Control of Single Intersection at Urban Traffic Network , 2010, IEEE Systems Journal.

[25]  Baher Abdulhai,et al.  Multi-Agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC) , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[26]  Kenneth Tze Kin Teo,et al.  Agent-Based Traffic Flow Optimization at Multiple Signalized Intersections , 2014, 2014 8th Asia Modelling Symposium.

[27]  Baher Abdulhai,et al.  An agent-based learning towards decentralized and coordinated traffic signal control , 2010, 13th International IEEE Conference on Intelligent Transportation Systems.

[28]  A.G. Sims,et al.  The Sydney coordinated adaptive traffic (SCAT) system philosophy and benefits , 1980, IEEE Transactions on Vehicular Technology.

[29]  Jie Wang,et al.  Traffic signal control with macroscopic fundamental diagrams , 2015, 2015 American Control Conference (ACC).

[30]  Christian Bettstetter,et al.  On the Message and Time Complexity of a Distributed Mobility – Adaptive Clustering Algorithm in Wireless Ad Hoc Networks , 2001 .

[31]  Xiaoliang Ma,et al.  Adaptive Group-based Signal Control by Reinforcement Learning☆ , 2015 .

[32]  Dipti Srinivasan,et al.  Cooperative, hybrid agent architecture for real-time traffic signal control , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[33]  Walid Gomaa,et al.  Freeway ramp-metering control based on Reinforcement learning , 2014, 11th IEEE International Conference on Control & Automation (ICCA).

[34]  Juan C. Medina,et al.  Traffic signal control using reinforcement learning and the max-plus algorithm as a coordinating strategy , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[35]  Monireh Abdoos,et al.  Traffic light control in non-stationary environments based on multi agent Q-learning , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[36]  J. Chinrungrueng,et al.  Performance Comparison between Queueing Theoretical Optimality and Q-Learning Approach for Intersection Traffic Signal Control , 2012, 2012 Fourth International Conference on Computational Intelligence, Modelling and Simulation.

[37]  Shalabh Bhatnagar,et al.  Threshold Tuning Using Stochastic Optimization for Graded Signal Control , 2012, IEEE Transactions on Vehicular Technology.

[38]  Vinny Cahill,et al.  Towards autonomic urban traffic control with collaborative multi-policy reinforcement learning , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[39]  Ali Hajbabaie,et al.  Arterial traffic control using reinforcement learning agents and information from adjacent intersections in the state and reward structure , 2010, 13th International IEEE Conference on Intelligent Transportation Systems.

[40]  Chen-Khong Tham,et al.  SensorGrid for Real-Time Traffic Management , 2007, 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information.

[41]  Jing Liu,et al.  Cooperative multi-agent traffic signal control system using fast gradient-descent function approximation for V2I networks , 2014, 2014 IEEE International Conference on Communications (ICC).

[42]  Shalabh Bhatnagar,et al.  Multi-agent reinforcement learning for traffic signal control , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[43]  T. Urbanik,et al.  Reinforcement learning-based multi-agent system for network traffic signal control , 2010 .

[44]  Carlos Gershenson,et al.  Self-organizing traffic lights: A realistic simulation , 2013 .

[45]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[46]  Shalabh Bhatnagar,et al.  Reinforcement Learning With Function Approximation for Traffic Signal Control , 2011, IEEE Transactions on Intelligent Transportation Systems.

[47]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[48]  P. Jain,et al.  Fuzzy Based Real Time Traffic Signal Controller to Optimize Congestion Delays , 2012, 2012 Second International Conference on Advanced Computing & Communication Technologies.

[49]  Ana L. C. Bazzan,et al.  Opportunities for multiagent systems and multiagent reinforcement learning in traffic control , 2009, Autonomous Agents and Multi-Agent Systems.

[50]  Paulo Martins Engel,et al.  Dealing with continuous-state reinforcement learning for intelligent control of traffic signals , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[51]  Carlos Gershenson,et al.  Self-organizing traffic lights: A realistic simulation , 2006, Advances in Applied Self-organizing Systems.

[52]  Meng Wang,et al.  Urban Traffic Signal Learning Control Using Fuzzy Actor-Critic Methods , 2009, ICNC.

[53]  Vinny Cahill,et al.  A Collaborative Reinforcement Learning Approach to Urban Traffic Control Optimization , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[54]  Leemon C. Baird,et al.  Residual Algorithms: Reinforcement Learning with Function Approximation , 1995, ICML.