Multiagent Systems: A Survey from a Machine Learning Perspective

Distributed Artificial Intelligence (DAI) has existed as a subfield of AI for less than two decades. DAI is concerned with systems that consist of multiple independent entities that interact in a domain. Traditionally, DAI has been divided into two sub-disciplines: Distributed Problem Solving (DPS) focuses on the information management aspects of systems with several components working together towards a common goal; Multiagent Systems (MAS) deals with behavior management in collections of several independent entities, or agents. This survey of MAS is intended to serve as an introduction to the field and as an organizational framework. A series of general multiagent scenarios are presented. For each scenario, the issues that arise are described along with a sampling of the techniques that exist to deal with them. The presented techniques are not exhaustive, but they highlight how multiagent systems can be and have been used to build complex systems. When options exist, the techniques presented are biased towards machine learning approaches. Additional opportunities for applying machine learning to MAS are highlighted and robotic soccer is presented as an appropriate test bed for MAS. This survey does not focus exclusively on robotic systems. However, we believe that much of the prior research in non-robotic MAS is relevant to robotic MAS, and we explicitly discuss several robotic MAS, including all of those presented in this issue.

[1]  Reid G. Smith,et al.  The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver , 1980, IEEE Transactions on Computers.

[2]  M. Benda,et al.  On Optimal Cooperation of Knowledge Sources , 1985 .

[3]  Keith S. Decker,et al.  Distributed problem-solving techniques: A survey , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[4]  Thomas Rist,et al.  On the Simultaneous Interpretation of Real World Image Sequences and their Natural Language Description: The System Soccer , 1988, ECAI.

[5]  Alan H. Bond,et al.  Distributed Artificial Intelligence , 1988 .

[6]  A. H. Bond.  An Analysis of Problems and Research in DAI , 1988 .

[7]  Edmund H. Durfee,et al.  Trends in Cooperative Distributed Problem Solving , 1989, IEEE Trans. Knowl. Data Eng..

[8]  G. Reeke.  The society of mind , 1991 .

[9]  Michael R. Genesereth,et al.  Knowledge Interchange Format , 1991, KR.

[10]  Edmund H. Durfee,et al.  What Your Computer Really Needs to Know, You Learned in Kindergarten , 1992, AAAI.

[11]  M. R. Genesereth,et al.  Knowledge Interchange Format Version 3.0 Reference Manual , 1992, LICS 1992.

[12]  Nicholas R. Jennings,et al.  ARCHON: theory and practice , 1992 .

[13]  Yoav Shoham,et al.  Agent-Oriented Programming , 1992, Artif. Intell..

[14]  Toshiharu Sugawara,et al.  On-Line Learning of Coordination Plans , 1993 .

[15]  Alan K. Mackworth.  On Seeing Robots , 1993, Computer Vision: Systems, Theory and Applications.

[16]  James J. Little,et al.  Dynamo: Real-time Experiments With Multiple Mobile Robots , 1993, Proceedings of the Intelligent Vehicles '93 Symposium.

[17]  Dean A. Pomerleau,et al.  Neural Network Perception for Mobile Robot Guidance , 1993 .

[18]  Maja J. Matarić,et al.  Leaning to behave socially , 1994 .

[19]  Claudia V. Goldman,et al.  Emergent Coordination through the Use of Cooperative State-Changing Rules , 1994, AAAI.

[20]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[21]  Tucker R. Balch,et al.  Communication in reactive multiagent robotic systems , 1995, Auton. Robots.

[22]  Maja J. Mataric,et al.  Interaction and intelligent behavior , 1994 .

[23]  Lynne E. Parker,et al.  Heterogeneous multi-robot cooperation , 1994 .

[24]  Jeffrey S. Rosenschein and Gilad Zlotkin.  Rules of Encounter , 1994 .

[25]  Michael K. Sahota.  Reactive Deliberation: An Architecture for Real-Time Intelligent Control in Dynamic Environments , 1994, AAAI.

[26]  M. Matarić.  Learning to Behave Socially , 1994 .

[27]  Minoru Asada,et al.  Coordination of multiple behaviors acquired by a vision-based reinforcement learning , 1994, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'94).

[28]  Timothy W. Finin,et al.  Kqml: an information and knowledge exchange protocol , 1994 .

[29]  Jeffrey S. Rosenschein,et al.  Coalition, Cryptography, and Stability: Mechanisms for Coalition Formation in Task Oriented Domains , 2018, AAAI.

[30]  Edmund H. Durfee,et al.  Blissful Ignorance: Knowing Just Enough to Coordinate Well , 1995, ICMAS.

[31]  Eiichi Osawa,et al.  A Metalevel Coordination Strategy for Reactive Cooperative Planning , 1995, ICMAS.

[32]  John J. Leggett,et al.  PARAgente: Exploring the Issues in Agent-Based User Interfaces , 1995, ICMAS.

[33]  John J. Grefenstette,et al.  A Coevolutionary Approach to Learning Sequential Decision Rules , 1995, ICGA.

[34]  Michael Wooldridge,et al.  Understanding the Emergence of Conventions in Multi-Agent Systems , 1995, ICMAS.

[35]  I. Noda.  Soccer Server: a simulator of RoboCup , 1995 .

[36]  Yves Demazeau,et al.  Exploiting Social Reasoning to Deal with Agency Level Inconsistency , 1995, ICMAS.

[37]  Victor Lesser,et al.  Environment Centered Analysis and Design of Coordination Mechanisms , 1996 .

[38]  Victor Lesser,et al.  Learning Organizational Roles in a Heterogeneous Multi-agent System , 1995 .

[39]  Victor R. Lesser,et al.  Issues in Automated Negotiation and Electronic Commerce: Extending the Contract Net Framework , 1997, ICMAS.

[40]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[41]  Manuela M. Veloso,et al.  Beating a Defender in Robotic Soccer: Memory-Based Learning of a Continuous Function , 1995, NIPS.

[42]  Kerstin Dautenhahn,et al.  Getting to know each other - Artificial social intelligence for autonomous robots , 1995, Robotics Auton. Syst..

[43]  Hector J. Levesque,et al.  Communicative Actions for Artificial Agents , 1997, ICMAS.

[44]  Sarit Kraus,et al.  Coordination without Communication: Experimental Validation of Focal Point Techniques , 1997, ICMAS.

[45]  Tucker R. Balch,et al.  Motor Schema-Based Formation Control for Multiagent Robot Teams , 1995, ICMAS.

[46]  Afsaneh Haddadi.  Towards a Pragmatic Theory of Interactions , 1995, ICMAS.

[47]  Sarit Kraus,et al.  Task Allocation Via Coalition Formation Among Autonomous Agents , 1995, IJCAI.

[48]  Edmund H. Durfee,et al.  Recursive Agent Modeling Using Limited Rationality , 1995, ICMAS.

[49]  Scott H. Clearwater,et al.  A Multi-Agent System for Controlling Building Environments , 1995, ICMAS.

[50]  Andreas Lux,et al.  Understanding Cooperation: An Agent's Perspective , 1997, ICMAS.

[51]  Cristiano Castelfranchi,et al.  Commitments: From Individual Intentions to Groups and Organizations , 1995, ICMAS.

[52]  Hideyuki Nakashima,et al.  Organic Programming for Multi-Agents , 1995, ICMAS.

[53]  Larry Bull,et al.  Evolution in Multi-agent Systems: Evolving Communicating Classifier Systems for Gait in a Quadrupedal Robot , 1995, ICGA.

[54]  Victor R. Lesser,et al.  Designing a Family of Coordination Algorithms , 1997, ICMAS.

[55]  Anand S. Rao,et al.  BDI Agents: From Theory to Practice , 1995, ICMAS.

[56]  Alan K. Mackworth,et al.  Real-time control of soccer-playing robots using off-board vision: the dynamite testbed , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[57]  Jeffrey S. Rosenschein,et al.  Time and the Prisoner's Dilemma , 2007, ICMAS.

[58]  Sandip Sen,et al.  Strongly Typed Genetic Programming in Evolving Cooperation Strategies , 1995, ICGA.

[59]  Milind Tambe.  Recursive Agent and Agent-Group Tracking in a Real-Time Dynamic Environment , 1995, ICMAS.

[60]  Victor R. Lesser,et al.  Coalition Formation among Bounded Rational Agents , 1995, IJCAI.

[61]  Barbara Hayes-Roth,et al.  Multiagent Collaboration in Directed Improvisation , 1997, ICMAS.

[62]  Richard K. Belew,et al.  Methods for Competitive Co-Evolution: Finding Opponents Worth Beating , 1995, ICGA.

[63]  Mark S. Fox,et al.  COOL: A Language for Describing Coordination in Multi Agent Systems , 1995, ICMAS.

[64]  Yongyuth Permpoontanalarp,et al.  Generalised Proof-Theory for Multi-Agent Autoepistemic Reasoning , 1995, ICMAS.

[65]  Gerhard Weiß,et al.  Distributed reinforcement learning , 1995, Robotics Auton. Syst..

[66]  Peter Stone.  Broad Learning from Narrow Training: A Case Study in Robotic Soccer , 1995 .

[67]  Edmund H. Durfee,et al.  Deciding When to Commit to Action During Observation-Based Coordination , 1995, ICMAS.

[68]  Moshe Tennenholtz,et al.  Adaptive Load Balancing: A Study in Multi-Agent Learning , 1994, J. Artif. Intell. Res..

[69]  Tad Hogg,et al.  Dilemmas in Computational Societies , 1995, ICMAS.

[70]  Sandip Sen,et al.  Evolving Beharioral Strategies in Predators and Prey , 1995, Adaption and Learning in Multi-Agent Systems.

[71]  Victor R. Lesser,et al.  Multiagent systems: an emerging subdiscipline of AI , 1995, CSUR.

[72]  Tuomas Sandholm,et al.  On Multiagent Q-Learning in a Semi-Competitive Domain , 1995, Adaption and Learning in Multi-Agent Systems.

[73]  Katia P. Sycara,et al.  Distributed Intelligent Agents , 1996, IEEE Expert.

[74]  Milind Tambe,et al.  Tracking Dynamic Team Activity , 1996, AAAI/IAAI, Vol. 1.

[75]  Sandip Sen,et al.  Effects of Local Information on Group Behavior , 1996, AAAI/IAAI, Vol. 2.

[76]  Michael R. M. Jenkin,et al.  A taxonomy for multi-agent robotics , 1996, Auton. Robots.

[77]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[78]  Nicholas R. Jennings,et al.  Foundations of distributed artificial intelligence , 1996, Sixth-generation computer technology series.

[79]  Hyungsuck Cho.  Multiple Micro Robots Playing Robot Soccer Game , 1996 .

[80]  Sandip Sen,et al.  Correlating Internal Parameters and External Performance: Learning Soccer Agents , 1996, ECAI Workshop LDAIS / ICMAS Workshop LIOME.

[81]  Sorin Achim and Peter Stone and Manuela Veloso.  Building a Dedicated Robotic Soccer System , 1996 .

[82]  John J. Grefenstette,et al.  Methods for Competitive and Cooperative Co-evolution , 1996 .

[83]  Xuemei Wang.  Planning While Learning Operators , 1996, AIPS.

[84]  Neeraj Arora,et al.  Resolving Social Dilemmas Using Genetic Algorithms , 1996 .

[85]  Innes A. Ferguson,et al.  Multiagent Learning and Adaptation in an Information Filtering Market , 1996 .

[86]  Sandip Sen.  IJCAI-95 Workshop on Adaptation and Learning in Multiagent Systems , 1996 .

[87]  Victor R. Lesser,et al.  Advantages of a Leveled Commitment Contracting Protocol , 1996, AAAI/IAAI, Vol. 1.

[88]  Minoru Asada,et al.  Action-based sensor space categorization for robot learning , 1996, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS '96.

[89]  Sandip Sen,et al.  Learning Cases to Compliment Rules for Conflict Resolution in Multiagent Systems , 1996 .

[90]  Keith Decker.  Distributed artificial intelligence testbeds , 1996 .

[91]  H. Van Dyke Parunak,et al.  Applications of distributed artificial intelligence in industry , 1996 .

[92]  Jong-Hwan Kim,et al.  Designing Distributed Control Architecture for Cooperative Multiagent System , 1996 .

[93]  Minoru Asada,et al.  Behavior coordination for a mobile robot using modular reinforcement learning , 1996, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS '96.

[94]  Jfirgen Schmidhuber,et al.  A GENERAL METHOD FOR MULTI-AGENT REINFORCEMENT LEARNING IN UNRESTRICTED ENVIRONMENTS , 1996 .

[95]  M. Ridley.  The Origins of Virtue , 1996 .

[96]  Yoichi Muraoka,et al.  A Method Applied for Soccer's Behaviors Using Proper Feedback and Feedforward Control , 1997, RoboCup.

[97]  Yaser Al-Onaizan,et al.  Using an Explicit Model of Teamwork in RoboCup-97 , 1997, RoboCup.

[98]  Hiroaki Kitano,et al.  RoboCup-97: Robot Soccer World Cup I , 1998, Lecture Notes in Computer Science.

[99]  Itsuki Noda.  Team GAMMA: Agent Programming on Gaea , 1997, RoboCup.

[100]  Katsushi Ikeuchi,et al.  Symbolic visual learning , 1997 .

[101]  Hiroaki Kitano,et al.  The RoboCup Synthetic Agent Challenge 97 , 1997, IJCAI.

[102]  Andrew Jennings,et al.  RoboCup97: An Omnidirectional Perspective , 1997, RoboCup.

[103]  James A. Hendler,et al.  Co-evolving Soccer Softbot Team Coordination with Genetic Programming , 1997, RoboCup.

[104]  Josep Puyol-Gruart,et al.  Soccer team based on agent-oriented programming , 1997, Robotics Auton. Syst..

[105]  Tomohito Andou,et al.  Refinement of Soccer Agents' Positions Using Reinforcement Learning , 1997, RoboCup.

[106]  Manuela M. Veloso,et al.  The CMUnited-97 Small Robot Team , 1997, RoboCup.

[107]  Andrew B. Kahng,et al.  Cooperative Mobile Robotics: Antecedents and Directions , 1997, Auton. Robots.

[108]  Hiroaki Kitano,et al.  RoboCup: The Robot World Cup Initiative , 1997, AGENTS '97.

[109]  Hajime Asama,et al.  Omni-directional Autonomous Robots Cooperating for Team Play , 1997, RoboCup.

[110]  Thomas Rist,et al.  Generating Multimedia Presentations for RoboCup Soccer Games , 1997, RoboCup.

[111]  Masahiro Fujita,et al.  An open architecture for robot entertainment , 1997, AGENTS '97.

[112]  Milind Tambe,et al.  Towards Flexible Teamwork , 1997, J. Artif. Intell. Res..

[113]  Akihiro Matsumoto,et al.  Decision Making by the Characteristics and the Interaction in Multi-agent Robotics Soccer , 1997, RoboCup.

[114]  Atsushi Shinjoh.  RoboCup-3D: The Construction of Intelligent Navigation System , 1997, RoboCup.

[115]  Tucker Balch,et al.  Learning Roles: Behavioral Diversity in Robot Teams , 1997 .

[116]  Jukka Riekki,et al.  Playing Soccer by Modifying and Combining Primitive Reactions , 1997, RoboCup.

[117]  S. F. Actory,et al.  Personal correspondence , 1997 .

[118]  Hiroaki Kitano,et al.  RoboCup: A Challenge Problem for AI , 1997, AI Mag..

[119]  Silvia Coradeschi,et al.  A Role-Based Decision-Mechanism for Teams of Reactive and Coordinating Agents , 1997, RoboCup.

[120]  Paul Scerri.  A Multi-Layered Behavior Based System for Controlling RoboCup Agents , 1997, RoboCup.

[121]  Richard K. Belew,et al.  New Methods for Competitive Coevolution , 1997, Evolutionary Computation.

[122]  Gordon Cheng,et al.  Real-Time Vision Processing for a Soccer Playing Mobile Robot , 1997, RoboCup.

[123]  Randy Sargent,et al.  Dynamic Object Capture Using Fast Vision Tracking , 1997, AI Mag..

[124]  Gerhard Weiß.  Distributed Artificial Intelligence Meets Machine Learning Learning in Multi-Agent Environments , 1997, Lecture Notes in Computer Science.

[125]  Kim Binsted,et al.  Character Design for Soccer Commentary , 1998, RoboCup.

[126]  北野 宏明,et al.  RoboCup-97 : robot soccer World Cup I , 1998 .

[127]  Manuela M. Veloso,et al.  Layered Approach to Learning Client Behaviors in the Robocup Soccer Server , 1998, Appl. Artif. Intell..

[128]  Ian Frank,et al.  Soccer Server: A Tool for Research on Multiagent Systems , 1998, Appl. Artif. Intell..

[129]  Vidroha Debroy,et al.  Genetic Programming , 1998, Lecture Notes in Computer Science.

[130]  Manuela M. Veloso,et al.  CMUNITED-97: RoboCup-97 Small-Robot World Champion Team , 1998, AI Mag..

[131]  Bernhard Nebel,et al.  The CS Freiburg Team , 1998 .

[132]  Katia P. Sycara,et al.  Bayesian learning in negotiation , 1998, Int. J. Hum. Comput. Stud..

[133]  Sandip Sen,et al.  Learning cases to resolve conflicts and improve group behavior , 1998, Int. J. Hum. Comput. Stud..

[134]  Michael J. Prietula,et al.  Simulating organizations: computational models of institutions and groups , 1998 .

[135]  M. Ridley.  The origins of virtue : human instincts and the evolution of cooperation , 1998 .

[136]  Peter Stone,et al.  Individual and collaborative behaviors in a team of homogeneous robotic soccer agents , 1998, Proceedings International Conference on Multi Agent Systems (Cat. No.98EX160).

[137]  Manuela M. Veloso,et al.  The CMTrio-98 Sony-Legged Robot Team , 1998, RoboCup.

[138]  R. Arkin,et al.  Behavioral diversity in learning robot teams , 1998 .

[139]  Manuela M. Veloso,et al.  The CMUnited-98 champion small-robot team , 1998, Adv. Robotics.

[140]  Lin Padgham,et al.  From Roles to Teamwork: A Framework and Architecture , 1998, Appl. Artif. Intell..

[141]  Peter Stone,et al.  CMUnited: a team of robotics soccer agents collaborating in an adversarial environment , 1998, CROS.

[142]  Astro Teller,et al.  Evolving Team Darwin United , 1998, RoboCup.

[143]  Victor R. Lesser,et al.  Learning organizational roles for negotiated search in a multiagent system , 1998, Int. J. Hum. Comput. Stud..

[144]  Hiroaki Kitano,et al.  Playing soccer with legged robots , 1998, Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No.98CH36190).

[145]  Manuela M. Veloso,et al.  The CMUnited-98 Champion Simulator Team , 1998, RoboCup.

[146]  Hiroaki Kitano,et al.  RoboCup-97: The First Robot World Cup Soccer Games and Conferences , 1998, AI Mag..

[147]  Manuela M. Veloso,et al.  Reactive visual control of multiple non-holonomic robotic agents , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[148]  H. Van Dyke Parunak,et al.  A systematic market approach to distributed constraint problems , 1998, Proceedings International Conference on Multi Agent Systems (Cat. No.98EX160).

[149]  Yoichi Muraoka,et al.  A Method Applied for Soccer's Behaviors Using Proper Feedback and Feedforward Control , 1998 .

[150]  Pattie Maes,et al.  Trafficopter: A Distributed Collection System for Traffic Information , 1998, CIA.

[151]  Hadi Moradi,et al.  Building integrated mobile robots for soccer competition , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[152]  Sandip Sen,et al.  Using limited information to enhance group stability , 1998, Int. J. Hum. Comput. Stud..

[153]  Sandip Sen,et al.  Evolution and learning in multiagent systems , 1998, Int. J. Hum. Comput. Stud..

[154]  Manuela M. Veloso,et al.  Towards collaborative and adversarial learning: a case study in robotic soccer , 1998, Int. J. Hum. Comput. Stud..

[155]  Kôiti Hasida,et al.  Automatic Soccer Commentary and RoboCup , 1998, RoboCup.

[156]  Keith S. Decker,et al.  Task environment centered simulation , 1998 .

[157]  Hiroaki Kitano,et al.  RoboCup: Robot World Cup , 1998, CROS.

[158]  Motion control in dynamic multi-robot environments , 1999, Proceedings 1999 IEEE International Symposium on Computational Intelligence in Robotics and Automation. CIRA'99 (Cat. No.99EX375).

[159]  Manuela M. Veloso,et al.  Motion Control in Dynamic Multi-Robot Environments , 1999, RoboCup.

[160]  Manuela M. Veloso,et al.  Task Decomposition, Dynamic Role Assignment, and Low-Bandwidth Communication for Real-Time Strategic Teamwork , 1999, Artif. Intell..

[161]  Christiaan J. J. Paredis,et al.  Heterogeneous Teams of Modular Robots for Mapping and Exploration , 2000, Auton. Robots.

[162]  Alexander Zelinsky,et al.  Grounded Symbolic Communication between Heterogeneous Cooperating Robots , 2000, Auton. Robots.

[163]  Tucker R. Balch,et al.  Hierarchic Social Entropy: An Information Theoretic Measure of Robot Group Diversity , 2000, Auton. Robots.

[164]  Lynne E. Parker,et al.  Lifelong Adaptation in Heterogeneous Multi-Robot Teams: Response to Continual Variation in Individual Robot Performance , 2000, Auton. Robots.

[165]  Peter Stone,et al.  Layered learning in multiagent systems - a winning approach to robotic soccer , 2000, Intelligent robotics and autonomous agents.

[166]  Wei-Min Shen,et al.  CONRO: Towards Deployable Robots with Inter-Robots Metamorphic Capabilities , 2000, Auton. Robots.

[167]  Jean-Louis Deneubourg,et al.  From local actions to global tasks: stigmergy and collective robotics , 2000 .

[168]  Wolfram Burgard,et al.  A Probabilistic Approach to Collaborative Multi-Robot Localization , 2000, Auton. Robots.

[169]  Hiroaki Kitano,et al.  RoboCup-98: Robot Soccer World Cup II , 2001, Lecture Notes in Computer Science.

[170]  O. Holland.  Multiagent systems : Lessons from social insects and collective robotics , 2002 .

[171]  Hitoshi Matsubara,et al.  Learning of Cooperative actions in multi-agent systems: a case study of pass play in Soccer , 2002 .