Soilse: A decentralized approach to optimization of fluctuating urban traffic using Reinforcement Learning

Increasing traffic congestion is a major problem in urban areas, which incurs heavy economic and environmental costs in both developing and developed countries. Efficient urban traffic control (UTC) can help reduce traffic congestion. However, the increasing volume and the dynamic nature of urban traffic pose particular challenges to UTC. Reinforcement Learning (RL) has been shown to be a promising approach to efficient UTC. However, most existing work on RL-based UTC does not adequately address the fluctuating nature of urban traffic. This paper presents Soilse1, a decentralized RL-based UTC optimization scheme that includes a nonparametric pattern change detection mechanism to identify local traffic pattern changes that adversely affect an RL agent's performance. Hence, Soilse is adaptive as agents learn to optimize for different traffic patterns and responsive as agents can detect genuine traffic pattern changes and trigger relearning. We compare the performance of Soilse to two baselines, a fixed-time approach and a saturation balancing algorithm that emulates SCATS, a well-known UTC system. The comparison was performed based on a simulation of traffic in Dublin's inner city centre. Results from using our scheme show an approximate 35%–43% and 40%–54% better performance in terms of average vehicle waiting time and average number of vehicle stops respectively against the best baseline performance in our simulation.

[1]  Bram Bakker,et al.  Reinforcement Learning of Traffic Light Controllers Adapting to Traffic Congestion , 2005, BNAIC.

[2]  Mark Humphreys,et al.  Action selection methods using reinforcement learning , 1997 .

[3]  Ana L. C. Bazzan,et al.  A Distributed Approach for Coordination of Traffic Signal Agents , 2005, Autonomous Agents and Multi-Agent Systems.

[4]  Vasilios A. Siris,et al.  Application of anomaly detection algorithms for detecting SYN flooding attacks , 2004, IEEE Global Telecommunications Conference, 2004. GLOBECOM '04..

[5]  Hartmut Schmeck,et al.  Organic Control of Traffic Lights , 2008, ATC.

[6]  Michael Schreckenberg,et al.  A cellular automaton model for freeway traffic , 1992 .

[7]  R W Denney,et al.  Signal Timing Under Saturated Conditions , 2008 .

[8]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[9]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[10]  Jörg Hähner,et al.  Decentralised Progressive Signal Systems for Organic Traffic Control , 2008, 2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing Systems.

[11]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[12]  R. D. Bretherton,et al.  Optimizing networks of traffic signals in real time-the SCOOT method , 1991 .

[13]  P R Lowrie,et al.  The Sydney coordinated adaptive traffic system - principles, methodology, algorithms , 1982 .

[14]  Vinny Cahill,et al.  Distributed W-Learning: Multi-Policy Optimization in Self-Organizing Systems , 2009, 2009 Third IEEE International Conference on Self-Adaptive and Self-Organizing Systems.

[15]  Sev V. Nagalingam,et al.  CIM Justification and Optimisation , 1999 .

[16]  Jin Yu,et al.  Natural Actor-Critic for Road Traffic Optimisation , 2006, NIPS.

[17]  Baher Abdulhai,et al.  Reinforcement learning for true adaptive traffic signal control , 2003 .

[18]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[19]  S. Lafortune,et al.  Dynamic traffic control: decentralized and coordinated methods , 1997, Proceedings of Conference on Intelligent Transportation Systems.

[20]  Jean-Loup Farges,et al.  THE PRODYN REAL TIME TRAFFIC ALGORITHM , 1983 .

[21]  Pitu B. Mirchandani,et al.  A REAL-TIME TRAFFIC SIGNAL CONTROL SYSTEM: ARCHITECTURE, ALGORITHMS, AND ANALYSIS , 2001 .

[22]  Xin Chen,et al.  Intelligent cooperation control of urban traffic networks , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[23]  Fei-Yue Wang,et al.  RHODES to Intelligent Transportation Systems , 2005, IEEE Intell. Syst..

[24]  Wang,et al.  Review of road traffic control strategies , 2003, Proceedings of the IEEE.

[25]  Equipment Corp,et al.  The Sydney Coordinated Adaptive Traffic (SCAT) System Philosophy and Benefits , 1980 .

[26]  Vinny Cahill,et al.  A Collaborative Reinforcement Learning Approach to Urban Traffic Control Optimization , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[27]  Tae Yoon Kim,et al.  Variance change point detection via artificial neural networks for data separation , 2005, Neurocomputing.

[28]  Dipti Srinivasan,et al.  Neural Networks for Real-Time Traffic Signal Control , 2006, IEEE Transactions on Intelligent Transportation Systems.

[29]  Vinny Cahill,et al.  Requirements for an ubiquitous computing simulation and emulation environment , 2006, InterSense '06.

[30]  Klaus Truemper,et al.  A Logic Programming Based Approach for On-Line Traffic Control , 2006 .

[31]  Nathan H. Gartner,et al.  OPAC: A DEMAND-RESPONSIVE STRATEGY FOR TRAFFIC SIGNAL CONTROL , 1983 .

[32]  Eduardo Camponogara,et al.  Distributed Learning Agents in Urban Traffic Control , 2003, EPIA.

[33]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[34]  Kang G. Shin,et al.  Detecting SYN flooding attacks , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[35]  Markos Papageorgiou,et al.  Applications of the urban traffic control strategy TUC , 2006, Eur. J. Oper. Res..

[36]  Angela Di Febbraro,et al.  Urban traffic control structure based on hybrid Petri nets , 2004, IEEE Transactions on Intelligent Transportation Systems.

[37]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[38]  Bart De Schutter,et al.  Optimal Traffic Light Control for a Single Intersection , 1999, Eur. J. Control.

[39]  Marco Wiering,et al.  Multi-Agent Reinforcement Learning for Traffic Light control , 2000 .

[40]  Peter Stone,et al.  Multiagent traffic management: a reservation-based intersection control mechanism , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..