Deep Reinforcement Learning for the Co-Optimization of Vehicular Flow Direction Design and Signal Control Policy for a Road Network

Reinforcement Learning (RL) is a popular approach for deciding on an optimum traffic signal control policy to alleviate congestion in a road network. However, the traffic signal control policy can also be optimized in conjunction with the design of vehicular flow directions to further improve traffic performance. The design of vehicular flow directions refers to the right of way or directional restriction imposed in a road network. Here, a new RL-based technique is presented for co-optimization of the design of vehicular flow directions and control policy for traffic signals. This technique consists of a two-step iterative process, wherein a set of vehicular flow directions for a road network is generated, then a RL-based approach is used to train the traffic signal control policy over the given set of vehicular flow directions. Following the proposed technique, the vehicular flow directions with poor traffic performance are iteratively eliminated, while new vehicular flow directions are generated to achieve better traffic performance and realize convergence to a maximum possible expected traffic performance. The proposed RL-based technique is evaluated by using two examples under rush hour and non-rush hour traffic conditions. It is found that, compared to a RL-based approach in which only traffic signal control policy is considered, the proposed approach can be used to obtain a better traffic performance in terms of vehicular queue length and throughput.

[1]  Ludovica Adacher,et al.  Performance Analysis of Decentralized VS Centralized Control for the Traffic Signal Synchronization Problem , 2020 .

[2]  Kenneth Tze Kin Teo,et al.  Q-Learning Based Traffic Optimization in Management of Signal Timing Plan , 2020 .

[3]  Qionghai Dai,et al.  Cooperative Deep Reinforcement Learning for Large-Scale Traffic Grid Signal Control , 2020, IEEE Transactions on Cybernetics.

[4]  Yasin Yilmaz,et al.  Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey , 2020, IEEE Transactions on Intelligent Transportation Systems.

[5]  Denis Larocque,et al.  IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control , 2020, IEEE Transactions on Intelligent Transportation Systems.

[6]  Komal Jagdale,et al.  Adaptive Traffic Control System using Reinforcement Learning , 2020 .

[7]  Heni Ben Amor,et al.  Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learning , 2019, CoRL.

[8]  Hali Pang,et al.  Deep Deterministic Policy Gradient for Traffic Signal Control of Single Intersection , 2019, 2019 Chinese Control And Decision Conference (CCDC).

[9]  Ching-Yao Chan,et al.  A Reinforcement Learning Approach for Intelligent Traffic Signal Control at Urban Intersections , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[10]  Tianshu Chu,et al.  Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control , 2019, IEEE Transactions on Intelligent Transportation Systems.

[11]  Zhu Han,et al.  A Deep Reinforcement Learning Network for Traffic Light Cycle Control , 2018, IEEE Transactions on Vehicular Technology.

[12]  Saiedeh N. Razavi,et al.  Asynchronous n-step Q-learning adaptive traffic signal control , 2019, J. Intell. Transp. Syst..

[13]  David Ha,et al.  Reinforcement Learning for Improving Agent Design , 2018, Artificial Life.

[14]  Zhenhui Li,et al.  IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control , 2018, KDD.

[15]  Minoru Ito,et al.  Bias Based General Framework for Delay Reduction in Backpressure Routing Algorithm , 2018, 2018 International Conference on Computing, Networking and Communications (ICNC).

[16]  M. Sharir,et al.  A strong-connectivity algorithm and its applications in data flow analysis. , 2018 .

[17]  Matthew R. Walter,et al.  Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[18]  Peter Corcoran,et al.  Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning , 2017, ArXiv.

[19]  Li Li,et al.  Traffic signal timing via deep reinforcement learning , 2016, IEEE/CAA Journal of Automatica Sinica.

[20]  Nicholas Madamopoulos,et al.  Managing traffic-light-duration by exploiting smart antenna technology (MATSAT) for coordinated multiple-intersections (CMI) , 2015, 2015 International Conference on Emerging Technologies (ICET).

[21]  Bart De Schutter,et al.  Co-design of traffic network topology and control measures , 2015 .

[22]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[23]  W. Y. Szeto,et al.  A bi-objective turning restriction design problem in urban road networks , 2014, Eur. J. Oper. Res..

[24]  Mohamed A. Khamis,et al.  Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework , 2014, Eng. Appl. Artif. Intell..

[25]  W. Y. Szeto,et al.  Multi-objective discrete urban road network design , 2013, Comput. Oper. Res..

[26]  Baher Abdulhai,et al.  Multiagent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC): Methodology and Large-Scale Application on Downtown Toronto , 2013, IEEE Transactions on Intelligent Transportation Systems.

[27]  W. Y. Szeto,et al.  Review on Urban Transportation Network Design Problems , 2013 .

[28]  Monireh Abdoos,et al.  Holonic multi-agent system for traffic signals control , 2013, Eng. Appl. Artif. Intell..

[29]  W. Y. Szeto,et al.  Hybrid Evolutionary Metaheuristics for Concurrent Multi-Objective Design of Urban Road and Public Transit Networks , 2012 .

[30]  Shalabh Bhatnagar,et al.  Reinforcement Learning With Function Approximation for Traffic Signal Control , 2011, IEEE Transactions on Intelligent Transportation Systems.

[31]  Hado van Hasselt,et al.  Double Q-learning , 2010, NIPS.

[32]  W. Y. Szeto,et al.  A turning restriction design problem in urban road networks , 2010, Eur. J. Oper. Res..

[33]  Dipti Srinivasan,et al.  Urban traffic signal control using reinforcement learning agents , 2010 .

[34]  Antonino Vitetta,et al.  The multi-criteria road network design problem in an urban area , 2006 .

[35]  H. Poorzahedy,et al.  Application of Ant System to network design problem , 2005 .

[36]  M E J Newman,et al.  Community structure in social and biological networks , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[37]  I. Mayeres,et al.  THE MARGINAL EXTERNAL COSTS OF URBAN TRANSPORT , 1996 .

[38]  Reid G. Simmons,et al.  Complexity Analysis of Real-Time Reinforcement Learning , 1993, AAAI.

[39]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[40]  R Bellman,et al.  DYNAMIC PROGRAMMING AND STATISTICAL COMMUNICATION THEORY. , 1957, Proceedings of the National Academy of Sciences of the United States of America.

[41]  Frans A. Oliehoek,et al.  Coordinated Deep Reinforcement Learners for Traffic Light Control , 2016 .

[42]  Peter Vrancx,et al.  Reinforcement Learning: State-of-the-Art , 2012 .

[43]  Takashi Oguchi,et al.  REDESIGN OF TRANSPORT SYSTEMS ON HIGHWAYS, STREETS AND AVENUES , 2008 .