Adaptive Traffic Signal Control: Deep Reinforcement Learning Algorithm with Experience Replay and Target Network

Adaptive traffic signal control, which adjusts traffic signal timing according to real-time traffic, has been shown to be an effective method to reduce traffic congestion. Available works on adaptive traffic signal control make responsive traffic signal control decisions based on human-crafted features (e.g. vehicle queue length). However, human-crafted features are abstractions of raw traffic data (e.g., position and speed of vehicles), which ignore some useful traffic information and lead to suboptimal traffic signal controls. In this paper, we propose a deep reinforcement learning algorithm that automatically extracts all useful features (machine-crafted features) from raw real-time traffic data and learns the optimal policy for adaptive traffic signal control. To improve algorithm stability, we adopt experience replay and target network mechanisms. Simulation results show that our algorithm reduces vehicle delay by up to 47% and 86% when compared to another two popular traffic signal control algorithms, longest queue first algorithm and fixed time control algorithm, respectively.

[1]  T. Urbanik,et al.  Reinforcement learning-based multi-agent system for network traffic signal control , 2010 .

[2]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[3]  Shalabh Bhatnagar,et al.  Reinforcement Learning With Function Approximation for Traffic Signal Control , 2011, IEEE Transactions on Intelligent Transportation Systems.

[4]  Li Li,et al.  Traffic signal timing via deep reinforcement learning , 2016, IEEE/CAA Journal of Automatica Sinica.

[5]  Daniel Krajzewicz,et al.  Recent Development and Applications of SUMO - Simulation of Urban MObility , 2012 .

[6]  Abdellah El Moudni,et al.  Approximate dynamic programming with recursive least-squares temporal difference learning for adaptive traffic signal control , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[7]  Itamar Elhanany,et al.  A Novel Signal-Scheduling Algorithm With Quality-of-Service Provisioning for an Isolated Intersection , 2008, IEEE Transactions on Intelligent Transportation Systems.

[8]  Dongbin Zhao,et al.  Computational Intelligence in Urban Traffic Signal Control: A Survey , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[9]  Frans A. Oliehoek,et al.  Video Demo: Deep Reinforcement Learning for Coordination in Traffic Light Control , 2016 .

[10]  Jim Duggan,et al.  An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control , 2016, Autonomic Road Transport Support Systems.

[11]  Emilio Frazzoli,et al.  Capacity-Aware Backpressure Traffic Signal Control , 2013, IEEE Transactions on Control of Network Systems.

[12]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[13]  Sagar Naik,et al.  Vehicular Networks for a Greener Environment: A Survey , 2013, IEEE Communications Surveys & Tutorials.

[14]  Saiedeh N. Razavi,et al.  Using a Deep Reinforcement Learning Agent for Traffic Signal Control , 2016, ArXiv.

[15]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[16]  Michael J. Neely,et al.  Dynamic power allocation and routing for satellite and wireless networks with time varying channels , 2003 .

[17]  Henk Wymeersch,et al.  Back-Pressure Traffic Signal Control With Fixed and Adaptive Routing for Urban Vehicular Networks , 2016, IEEE Transactions on Intelligent Transportation Systems.