Deep reinforcement learning for traffic signal control under disturbances: A case study on Sunway city, Malaysia

Abstract In most urban areas, traffic congestion is a vexing, complex and growing issue day by day. Reinforcement learning (RL) enables a single decision maker (or an agent) to learn and make optimal actions in an independent manner, while multi-agent reinforcement learning (MARL) enables multiple agents to exchange knowledge, learn, and make optimal joint actions in a collaborative manner. The integration of the newly emerging deep learning and the traditional RL approach has created an advanced technique called deep Q -network (DQN) that has shown promising results in solving high-dimensional and complex problems, including traffic congestion. In this paper, DQN is embedded in traffic signal control to solve traffic congestion issue, which has been plagued with the curse of dimensionality whereby the representation of the operating environment can be highly dimensional and complex when the traditional RL approach is used. Most importantly, this paper proposes multi-agent DQN (MADQN) and investigates its use to further address the curse of dimensionality under traffic network scenarios with high traffic volume and disturbances. To investigate the effectiveness of our proposed scheme, a case study based on an urban area, namely Sunway city in Malaysia, is conducted. We evaluate our scheme via simulation using a traffic network simulator called simulation of urban mobility (SUMO) and a simulation tool called MATLAB. Simulation results show that our proposed scheme reduces the total travel time of the vehicles.

[1]  Daniel Krajzewicz,et al.  SUMO - Simulation of Urban MObility An Overview , 2011 .

[2]  Carlos Gershenson,et al.  Self-organizing traffic lights: A realistic simulation , 2013 .

[3]  Jim Duggan,et al.  An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control , 2016, Autonomic Road Transport Support Systems.

[4]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[5]  Chia-Hao Wan,et al.  Value‐based deep reinforcement learning for adaptive isolated intersection signal control , 2018, IET Intelligent Transport Systems.

[6]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[7]  Peter Corcoran,et al.  Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning , 2017, ArXiv.

[8]  Mee Hong Ling,et al.  A Survey on Reinforcement Learning Models and Algorithms for Traffic Signal Control , 2017, ACM Comput. Surv..

[9]  Li Li,et al.  Traffic signal timing via deep reinforcement learning , 2016, IEEE/CAA Journal of Automatica Sinica.

[10]  Zhihua Zhang,et al.  Multivariate Time Series Analysis in Climate and Environmental Research , 2017 .

[11]  Abbas Khosravi,et al.  Q-learning method for controlling traffic signal phase time in a single intersection , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[12]  Zhenhui Li,et al.  IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control , 2018, KDD.

[13]  Shalabh Bhatnagar,et al.  Threshold Tuning Using Stochastic Optimization for Graded Signal Control , 2012, IEEE Transactions on Vehicular Technology.

[14]  Shalabh Bhatnagar,et al.  Decentralized learning for traffic signal control , 2015, 2015 7th International Conference on Communication Systems and Networks (COMSNETS).

[15]  W. W. Mosher,et al.  NEW STATISTICAL METHOD FOR DESCRIBING HIGHWAY DISTRIBUTION OF CARS , 1961 .

[16]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[17]  Johnnie Ben-Edigbe,et al.  Effect of Rain on Probability Distributions Fitted to Vehicle Time Headways , 2012 .

[18]  Winifred D. Ashton.  Distributions for Gaps in Road Traffic , 1971 .

[19]  Abdellah El Moudni,et al.  Traffic network micro-simulation model and control algorithm based on approximate dynamic programming , 2016 .

[20]  Shalabh Bhatnagar,et al.  Multi-agent reinforcement learning for traffic signal control , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[21]  Walid Gomaa,et al.  Multi-objective traffic light control system based on Bayesian probability interpretation , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[22]  Akhilesh Kumar Maurya,et al.  Speed and Time Headway Distribution under Mixed Traffic Condition , 2015 .

[23]  Wei Wang,et al.  Reinforcement Learning-Based Variable Speed Limit Control Strategy to Reduce Traffic Congestion at Freeway Recurrent Bottlenecks , 2017, IEEE Transactions on Intelligent Transportation Systems.

[24]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[25]  Alan J. Miller.  A Queueing Model for Road Traffic Flow , 1961 .

[26]  Jian Lin,et al.  Studies on Hierarchical Reinforcement Learning in Multi-Agent Environment , 2008, 2008 IEEE International Conference on Networking, Sensing and Control.

[27]  A. B. Elistina,et al.  Comparative Study on Lightning Fatality Rate in Malaysia between 2008 and 2017 , 2018, 2018 34th International Conference on Lightning Protection (ICLP).