MaCAR: Urban Traffic Light Control via Active Multi-agent Communication and Action Rectification

Urban traffic light control is an important and challenging real-world problem. By regarding intersections as agents, most of the Reinforcement Learning (RL) based methods generate actions of agents independently. They can cause action conflict and result in overflow or road resource waste in adjacent intersections. Recently, some collaborative methods have alleviated the above problems by extending the observable surroundings of agents, which can be considered as inactive cross-agent communication methods. However, when agents act synchronously in these works, the perceived action value is biased and the information exchanged is insufficient. In this work, we propose a novel Multi-agent Communication and Action Rectification (MaCAR) framework. It enables active communication between agents by considering the impact of synchronous actions of agents. MaCAR consists of two parts: (1) an active Communication Agent Network (CAN) involving a Message Propagation Graph Neural Network (MPGNN); (2) a Traffic Forecasting Network (TFN) which learns to predict the traffic after agents’ synchronous actions and the corresponding action values. By using predicted information, we mitigate the action value bias during training to help rectify agents’ future actions. In experiments, we show that our proposal can outperforms state-of-the-art methods on both synthetic and real-world datasets.

[1]  Frans A. Oliehoek,et al.  Coordinated Deep Reinforcement Learners for Traffic Light Control , 2016 .

[2]  Tianshu Chu,et al.  Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control , 2019, IEEE Transactions on Intelligent Transportation Systems.

[3]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[4]  Nan Xu,et al.  CoLight: Learning Network-level Cooperation for Traffic Signal Control , 2019, CIKM.

[5]  Xian-Sheng Hua,et al.  Dual Graph for Traffic Forecasting , 2019, IEEE Access.

[6]  Carlos Gershenson,et al.  Self-organizing traffic lights: A realistic simulation , 2006, Advances in Applied Self-organizing Systems.

[7]  Dahua Lin,et al.  Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition , 2018, AAAI.

[8]  Marco Wiering,et al.  Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events , 2017 .

[9]  Takayoshi Yoshimura,et al.  Traffic Signal Control Based on Reinforcement Learning with Graph Convolutional Neural Nets , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[10]  Nan Xu,et al.  Diagnosing Reinforcement Learning for Traffic Signal Control , 2019, ArXiv.

[11]  Kai Xu,et al.  PressLight: Learning Max Pressure Control to Coordinate Traffic Signals in Arterial Network , 2019, KDD.

[12]  Zhanxing Zhu,et al.  ST-UNet: A Spatio-Temporal U-Network for Graph-structured Time Series Modeling , 2019, ArXiv.

[13]  Zhenhui Li,et al.  IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control , 2018, KDD.

[14]  Volkan Cevher,et al.  Optimization for Reinforcement Learning: From a single agent to cooperative agents , 2020, IEEE Signal Processing Magazine.

[15]  Ming Yang,et al.  Toward A Thousand Lights: Decentralized Deep Reinforcement Learning for Large-Scale Traffic Signal Control , 2020, AAAI.

[16]  Kenneth Tze Kin Teo,et al.  Q-Learning Traffic Signal Optimization within Multiple Intersections Traffic Network , 2012, 2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation.

[17]  Noe Casas,et al.  Deep Deterministic Policy Gradient for Urban Traffic Light Control , 2017, ArXiv.

[18]  Cyrus Shahabi,et al.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting , 2017, ICLR.

[19]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.