Opportunities for multiagent systems and multiagent reinforcement learning in traffic control

The increasing demand for mobility in our society poses various challenges to traffic engineering, computer science in general, and artificial intelligence and multiagent systems in particular. As it is often the case, it is not possible to provide additional capacity, so that a more efficient use of the available transportation infrastructure is necessary. This relates closely to multiagent systems as many problems in traffic management and control are inherently distributed. Also, many actors in a transportation system fit very well the concept of autonomous agents: the driver, the pedestrian, the traffic expert; in some cases, also the intersection and the traffic signal controller can be regarded as an autonomous agent. However, the “agentification” of a transportation system is associated with some challenging issues: the number of agents is high, typically agents are highly adaptive, they react to changes in the environment at individual level but cause an unpredictable collective pattern, and act in a highly coupled environment. Therefore, this domain poses many challenges for standard techniques from multiagent systems such as coordination and learning. This paper has two main objectives: (i) to present problems, methods, approaches and practices in traffic engineering (especially regarding traffic signal control); and (ii) to highlight open problems and challenges so that future research in multiagent systems can address them.

[1]  Ana L. C. Bazzan,et al.  To Adapt or Not to Adapt - Consequences of Adapting Driver and Traffic Light Agents , 2007, Adaptive Agents and Multi-Agents Systems.

[2]  Dit-Yan Yeung,et al.  Hidden-Mode Markov Decision Processes for Nonstationary Sequential Decision Making , 2001, Sequence Learning.

[3]  Ana L. C. Bazzan,et al.  Case Studies on the Braess Paradox: Simulating Route Recommendation and Learning in Abstract and Microscopic Models , 2005 .

[4]  Alexis Drogoul,et al.  How to Combine Reactivity and Anticipation: The Case of Conflicts Resolution in a Simulated Road Traffic , 2000, MABS.

[5]  Yoav Shoham,et al.  Multi-Agent Reinforcement Learning:a critical survey , 2003 .

[6]  Ana L. C. Bazzan,et al.  Learning in groups of traffic signals , 2010, Eng. Appl. Artif. Intell..

[7]  Wilhelm Leutzbach,et al.  Introduction to the Theory of Traffic Flow , 1987 .

[8]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[9]  Iisakki Kosonen,et al.  Multi-agent fuzzy signal control based on real-time simulation , 2001 .

[10]  Michael Brady,et al.  Towards a behavioural traffic monitoring system , 2005, AAMAS '05.

[11]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[12]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[13]  Kenneth G Courage,et al.  TRANSYT-7F USER'S MANUAL , 1984 .

[14]  H. Pietrantonio.  URBAN TRAVEL DEMAND MODELING: FROM INDIVIDUAL CHOICES TO GENERAL EQUILIBRIUM , 1997 .

[15]  Sean Luke,et al.  History-based traffic control , 2006, AAMAS '06.

[16]  Nathan H. Gartner,et al.  OPAC: A DEMAND-RESPONSIVE STRATEGY FOR TRAFFIC SIGNAL CONTROL , 1983 .

[17]  Ana L. C. Bazzan,et al.  A Swarm-Based Approach for Selection of Signal Plans in Urban Scenarios , 2004, ANTS Workshop.

[18]  Jean-Loup Farges,et al.  THE PRODYN REAL TIME TRAFFIC ALGORITHM , 1983 .

[19]  Ana L. C. Bazan.  A Game-Theoretic Approach to Distributed Control of Traffic Signals , 1995, ICMAS.

[20]  Peter Stone,et al.  Multiagent traffic management: an improved intersection control mechanism , 2005, AAMAS '05.

[21]  P R Lowrie,et al.  The Sydney coordinated adaptive traffic system - principles, methodology, algorithms , 1982 .

[22]  Hartmut Schmeck,et al.  An Organic Architecture for Traffic Light Controllers , 2006, GI Jahrestagung.

[23]  Paulo Martins Engel,et al.  Dealing with non-stationary environments using context detection , 2006, ICML.

[24]  Yoav Shoham,et al.  If multi-agent learning is the answer, what is the question? , 2007, Artif. Intell..

[25]  Peter Stone,et al.  Sharing the Road: Autonomous Vehicles Meet Human Drivers , 2007, IJCAI.

[26]  Michael Schreckenberg,et al.  A cellular automaton model for freeway traffic , 1992 .

[27]  Eduardo Camponogara,et al.  Distributed Learning Agents in Urban Traffic Control , 2003, EPIA.

[28]  Ana L. C. Bazzan,et al.  Congestion tolls as utility alignment between agent and system optimum , 2006, AAMAS '06.

[29]  Dietrich Braess,et al.  Über ein Paradoxon aus der Verkehrsplanung , 1968, Unternehmensforschung.

[30]  Rosaldo J. F. Rossetti,et al.  Using BDI agents to improve driver modelling in a commuter scenario , 2002 .

[31]  Ana L. C. Bazzan,et al.  A Distributed Approach for Coordination of Traffic Signal Agents , 2004, Autonomous Agents and Multi-Agent Systems.

[32]  Ana L. C. Bazzan,et al.  Adaptation in Games with Many Co-evolving Agents , 2007, EPIA Workshops.

[33]  Manuela M. Veloso,et al.  Rational and Convergent Learning in Stochastic Games , 2001, IJCAI.

[34]  Randolph W. Hall,et al.  Handbook of transportation science , 1999 .

[35]  Andrew W. Moore,et al.  Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time , 1993, Machine Learning.

[36]  Simon Parsons,et al.  What evolutionary game theory tells us about multiagent learning , 2007, Artif. Intell..

[37]  Bram Bakker,et al.  Reinforcement Learning of Traffic Light Controllers Adapting to Traffic Congestion , 2005, BNAIC.

[38]  Yoav Shoham,et al.  Learning against multiple opponents , 2006, AAMAS '06.

[39]  Ana L. C. Bazzan,et al.  Traffic Lights Control with Adaptive Group Formation Based on Swarm Intelligence , 2006, ANTS Workshop.

[40]  John D. C. Little,et al.  SYNCHRONIZING TRAFFIC SIGNALS FOR MAXIMAL BANDWIDTH , 1964 .

[41]  A. Schadschneider,et al.  Statistical physics of vehicular traffic and some related systems , 2000, cond-mat/0007053.

[42]  Kagan Tumer,et al.  A Survey of Collectives , 2004 .

[43]  J Y Luk,et al.  TRANSYT: traffic network study tool , 1990 .

[44]  A Schadschneider,et al.  Optimizing traffic lights in a cellular automaton model for city traffic. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[45]  Wang,et al.  Review of road traffic control strategies , 2003, Proceedings of the IEEE.

[46]  G. Robinson.  Regulation of division of labor in insect societies. , 1992, Annual review of entomology.

[47]  Richard S. Sutton,et al.  Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming , 1990, ML.

[48]  Ana L. C. Bazzan.  An evolutionary game-theoretical approach for coordination of traffic signal agents , 1997 .

[49]  J G Wardrop,et al.  CORRESPONDENCE. SOME THEORETICAL ASPECTS OF ROAD TRAFFIC RESEARCH. , 1952 .

[50]  Dirk Helbing,et al.  Coherent moving states in highway traffic , 1998, Nature.

[51]  Peter Stone,et al.  Multiagent Traffic Management: Opportunities for Multiagent Learning , 2005, LAMAS.

[52]  Rolf H. Möhring,et al.  A MODEL AND FAST OPTIMIZATION METHOD FOR SIGNAL COORDINATION IN A NETWORK , 2006 .

[53]  Victor R. Lesser,et al.  Using cooperative mediation to coordinate traffic lights: a case study , 2005, AAMAS '05.

[54]  Larry Bull,et al.  Towards distributed adaptive control for road traffic junction signals using learning classifier systems , 2004 .

[55]  Mitsuo Kawato,et al.  Multiple Model-Based Reinforcement Learning , 2002, Neural Computation.

[56]  Andreas Schadschneider,et al.  Self-organization of traffic jams in cities: effects of stochastic dynamics and signal periods , 1999 .

[57]  Tom Lenaerts,et al.  An Evolutionary Game Theoretic Perspective on Learning in Multi-Agent Systems , 2004, Synthese.

[58]  Michael P. Wellman,et al.  Multiagent Reinforcement Learning: Theoretical Framework and an Algorithm , 1998, ICML.

[59]  F. Kluegl,et al.  Decision dynamics in a traffic scenario , 2000 .

[60]  William R. McShane,et al.  A review of pedestrian safety models for urban areas in Low and Middle Income Countries , 2016 .

[61]  D. Gordon.  The Organization of Work in Social Insect , 2003 .

[62]  Peter Stone,et al.  Learning and Multiagent Reasoning for Autonomous Agents , 2007, IJCAI.

[63]  Ana L. C. Bazzan,et al.  Selection of information types based on personal utility: a testbed for traffic information markets , 2003, AAMAS '03.

[64]  Markos Papageorgiou,et al.  A Multivariable Regulator Approach to Traffic-Responsive Network-Wide Signal Control , 2000 .

[65]  Rosaldo J. F. Rossetti,et al.  A Dynamic Network Simulation Model Based on Multi-Agent Systems , 2005, Applications of Agent Technology in Traffic and Transportation.

[66]  D. Gordon.  The organization of work in social insect colonies , 1996, Nature.

[67]  Sascha Ossowski,et al.  Designing Multiagent Decision Support Systems for Traffic Management , 2005, Applications of Agent Technology in Traffic and Transportation.

[68]  Ali A. Ghorbani,et al.  A multiagent system for optimizing urban traffic , 2003, IEEE/WIC International Conference on Intelligent Agent Technology, 2003. IAT 2003..

[69]  Maarten Peeters,et al.  Multi-agent Reinforcement Learning in Stochastic Single and Multi-stage Games , 2005, Adaptive Agents and Multi-Agent Systems.

[70]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[71]  Peter Stone,et al.  Multiagent learning is not the answer. It is the question , 2007, Artif. Intell..

[72]  Bart De Schutter,et al.  A Test Bed for Multi-Agent Control Systems in Road Traffic Management , 2005, Applications of Agent Technology in Traffic and Transportation.

[73]  Ana L. C. Bazzan,et al.  Re-routing Agents in an Abstract Traffic Scenario , 2008, SBIA.

[74]  Eugénio C. Oliveira,et al.  Learning from multiple sources , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[75]  Kamalakar Karlapalem,et al.  Multi agent simulation of unorganized traffic , 2002, AAMAS '02.

[76]  Birgit Burmeister,et al.  Agent-oriented traffic simulation , 1997 .

[77]  J. G. Wardrop,et al.  Some Theoretical Aspects of Road Traffic Research , 1952 .

[78]  Karl Tuyls,et al.  An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games , 2005, Autonomous Agents and Multi-Agent Systems.

[79]  Kai Nagel,et al.  Towards truly agent-based traffic and mobility simulations , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[80]  R D Bretherton,et al.  SCOOT-a Traffic Responsive Method of Coordinating Signals , 1981 .

[81]  Sean Luke,et al.  Cooperative Multi-Agent Learning: The State of the Art , 2005, Autonomous Agents and Multi-Agent Systems.

[82]  Ana L. C. Bazzan,et al.  Simulated Route Decision Behaviour: Simple Heuristics and Adaptation , 2004 .

[83]  Rolf H. Möhring,et al.  Minimizing Total Delay in Fixed-Time Controlled Traffic Networks , 2004, OR.

[84]  Marco Wiering,et al.  Multi-Agent Reinforcement Learning for Traffic Light control , 2000 .

[85]  Ana L. C. Bazzan,et al.  The impact of real-time information in a two-route scenario using agent-based simulation , 2002 .

[86]  Peter Stone,et al.  Multiagent traffic management: a reservation-based intersection control mechanism , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[87]  D I Robertson,et al.  TRANSYT: A TRAFFIC NETWORK STUDY TOOL , 1969 .

[88]  Ana L. C. Bazzan,et al.  Anticipatory Traffic Forecast Using Multi-Agent Techniques , 2000 .

[89]  Kagan Tumer,et al.  Aligning social welfare and agent preferences to alleviate traffic congestion , 2008, AAMAS.

[90]  Ana L. C. Bazzan,et al.  Agents in Traffic Modelling - From Reactive to Social Behaviour , 1999, KI.

[91]  Carlos Gershenson,et al.  Self-organizing Traffic Lights , 2004, Complex Syst..