Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility

ABSTRACT This research applies R-Markov Average Reward Technique based reinforcement learning (RL) algorithm, namely RMART, for vehicular signal control problem leveraging information sharing among signal controllers in connected vehicle environment. We implemented the algorithm in a network of 18 signalized intersections and compare the performance of RMART with fixed, adaptive, and variants of the RL schemes. Results show significant improvement in system performance for RMART algorithm with information sharing over both traditional fixed signal timing plans and real time adaptive control schemes. The comparison with reinforcement learning algorithms including Q learning and SARSA indicate that RMART performs better at higher congestion levels. Further, a multi-reward structure is proposed that dynamically adjusts the reward function with varying congestion states at the intersection. Finally, the results from test networks show significant reduction in emissions (CO, CO2, NOx, VOC, PM10) when RL algorithms are implemented compared to fixed signal timings and adaptive schemes.

[1]  Jean-Loup Farges,et al.  THE PRODYN REAL TIME TRAFFIC ALGORITHM , 1983 .

[2]  Shimon Whiteson,et al.  Multiagent Reinforcement Learning for Urban Traffic Control Using Coordination Graphs , 2008, ECML/PKDD.

[3]  A. Koopman,et al.  Simulation and optimization of traffic in a city , 2004, IEEE Intelligent Vehicles Symposium, 2004.

[4]  Itamar Elhanany,et al.  A Novel Signal-Scheduling Algorithm With Quality-of-Service Provisioning for an Isolated Intersection , 2008, IEEE Transactions on Intelligent Transportation Systems.

[5]  Baher Abdulhai,et al.  Towards multi-agent reinforcement learning for integrated network of optimal traffic controllers (MARLIN-OTC) , 2010 .

[6]  Abhijit Gosavi,et al.  Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning , 2003 .

[7]  Mashrur Chowdhury,et al.  An integrated modeling approach for facilitating emission estimations of alternative fueled vehicles , 2012 .

[8]  D. Schrank,et al.  2015 Urban Mobility Scorecard , 2015 .

[9]  Saiedeh Razavi,et al.  Impact of Connected Vehicle on Work Zone Network Safety through Dynamic Route Guidance , 2016 .

[10]  Ana L. C. Bazzan,et al.  A Distributed Approach for Coordination of Traffic Signal Agents , 2005, Autonomous Agents and Multi-Agent Systems.

[11]  Shalabh Bhatnagar,et al.  Reinforcement Learning With Function Approximation for Traffic Signal Control , 2011, IEEE Transactions on Intelligent Transportation Systems.

[12]  S. Bottoms.  Utopia , 2013 .

[13]  Byungkyu Brian Park,et al.  Development and Evaluation of a Cooperative Vehicle Intersection Control Algorithm Under the Connected Vehicles Environment , 2012, IEEE Transactions on Intelligent Transportation Systems.

[14]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[15]  Thomas L. Thorpe.  Vehicle Traffic Light Control Using SARSA , 1997 .

[16]  Jia Hu,et al.  Coordinated transit signal priority supporting transit progression under Connected Vehicle Technology , 2015 .

[17]  Markos Papageorgiou,et al.  A multivariable regulator approach to traffic-responsive network-wide signal control , 2000 .

[18]  Rainer Wiedemann,et al.  SIMULATION DES STRASSENVERKEHRSFLUSSES. , 1974 .

[19]  Pitu B. Mirchandani,et al.  A REAL-TIME TRAFFIC SIGNAL CONTROL SYSTEM: ARCHITECTURE, ALGORITHMS, AND ANALYSIS , 2001 .

[20]  Rahim F Benekohal,et al.  Reinforcement Learning Agents for Traffic Signal Control in Oversaturated Networks , 2011 .

[21]  N. H. C. Yung,et al.  A Multiple-Goal Reinforcement Learning Method for Complex Vehicle Overtaking Maneuvers , 2011, IEEE Transactions on Intelligent Transportation Systems.

[22]  Yukinori Kakazu,et al.  Genetic reinforcement learning for cooperative traffic signal control , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[23]  Aleksandar Stevanovic,et al.  Retiming Traffic Signals to Minimize Surrogate Safety Measures on , 2012 .

[24]  Yu-Chee Tseng,et al.  Dynamic Traffic Control with Fairness and Throughput Optimization Using Vehicular Communications , 2013, IEEE Journal on Selected Areas in Communications.

[25]  Charles Desjardins,et al.  Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach , 2011, IEEE Transactions on Intelligent Transportation Systems.

[26]  Stephan Olariu,et al.  A survey of vehicular communications for traffic signal optimization , 2015, Veh. Commun..

[27]  Yining Li,et al.  A Recurrent Neural Network Approach to Network-wide Traffic Signal Control , 2010 .

[28]  Hidenori Ishihara,et al.  Traffic signal networks simulator using emotional algorithm with individuality , 2001, ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585).

[29]  Baher Abdulhai,et al.  Reinforcement learning: Introduction to theory and potential for transport applications , 2003 .

[30]  Adel W. Sadek,et al.  Assessing the Mobility and Environmental Benefits of Reservation-Based Intelligent Intersections Using an Integrated Simulator , 2012, IEEE Transactions on Intelligent Transportation Systems.

[31]  Dipti Srinivasan,et al.  Neural Networks for Real-Time Traffic Signal Control , 2006, IEEE Transactions on Intelligent Transportation Systems.

[32]  Baher Abdulhai,et al.  Design of Reinforcement Learning Parameters for Seamless Application of Adaptive Traffic Signal Control , 2014, J. Intell. Transp. Syst..

[33]  John N. Tsitsiklis,et al.  On Average Versus Discounted Reward Temporal-Difference Learning , 2002, Machine Learning.

[34]  Wei-Song Lin,et al.  Metro Traffic Regulation by Adaptive Optimal Control , 2011, IEEE Transactions on Intelligent Transportation Systems.

[35]  T. Urbanik,et al.  Reinforcement learning-based multi-agent system for network traffic signal control , 2010 .

[36]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[37]  Prasad Tadepalli,et al.  Model-Based Average Reward Reinforcement Learning , 1998, Artif. Intell..

[38]  Baher Abdulhai,et al.  Reinforcement learning for true adaptive traffic signal control , 2003 .

[39]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[40]  Chi-Kwong Li,et al.  An approach to tune fuzzy controllers based on reinforcement learning for autonomous vehicle control , 2005, IEEE Transactions on Intelligent Transportation Systems.

[41]  Hesham Rakha,et al.  Calibration of Steady-State Car-Following Models Using Macroscopic Loop Detector Data , 2010 .

[42]  Ella Bingham.  Reinforcement learning in neurofuzzy traffic signal control , 2001, Eur. J. Oper. Res..

[43]  P R Lowrie,et al.  The Sydney coordinated adaptive traffic system - principles, methodology, algorithms , 1982 .

[44]  Sophie Midenet,et al.  The real-time urban traffic control system CRONOS: Algorithm and experiments , 2006 .

[45]  Song Bai,et al.  Integration of MOVES and dynamic traffic assignment models for fine-grained transportation and air quality analyses , 2011, 2011 IEEE Forum on Integrated and Sustainable Transportation Systems.

[46]  Ana L. C. Bazzan,et al.  Learning in groups of traffic signals , 2010, Eng. Appl. Artif. Intell..

[47]  Nathan H. Gartner,et al.  OPAC: A DEMAND-RESPONSIVE STRATEGY FOR TRAFFIC SIGNAL CONTROL , 1983 .

[48]  Tao Li,et al.  Adaptive Dynamic Programming for Multi-intersections Traffic Signal Intelligent Control , 2008, 2008 11th International IEEE Conference on Intelligent Transportation Systems.

[49]  Abhijit Gosavi,et al.  Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning , 2003 .

[50]  Monireh Abdoos,et al.  Traffic light control in non-stationary environments based on multi agent Q-learning , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[51]  Yiheng Feng,et al.  A real-time adaptive signal control in a connected vehicle environment , 2015 .

[52]  R D Bretherton,et al.  THE SCOOT ON-LINE TRAFFIC SIGNAL OPTIMISATION TECHNIQUE , 1982 .

[53]  Jaeyoung Kwak,et al.  Evaluating the impacts of urban corridor traffic signal optimization on vehicle emissions and fuel consumption , 2012 .

[54]  Zhang Yi,et al.  Multiobjective Reinforcement Learning for Traffic Signal Control Using Vehicular Ad Hoc Network , 2010, EURASIP J. Adv. Signal Process..

[55]  Dipti Srinivasan,et al.  Urban traffic signal control using reinforcement learning agents , 2010 .

[56]  Li-Wen Chen,et al.  Traffic Signal Optimization with Greedy Randomized Tabu Search Algorithm , 2012 .

[57]  Yan Li,et al.  Urban Traffic Signal Control Network Partitioning Using Self-Organizing Maps , 2011 .