Metrics for Multiagent Systems

A Multiagent System (MAS) is a software paradigm for building large scale intelligent distributed systems. Increasingly these systems are being deployed on handheld computing devices that rely on non-traditional communications mediums such as mobile ad hoc networks and satellite links. These systems present new challenges for computer scientists in describing system performance and analyzing competing systems. This chapter surveys existing metrics that can be used to describe MASs and related components. A framework for analyzing MASs is provided and an example of how this framework might be employed is given for the domain of distributed constraint reasoning.

[1]  Michael L. Anderson.  A Flexible Approach to Quantifying Various Dimensions of Environmental Complexity , 2004 .

[2]  Makoto Yokoo,et al.  Constant Cost of the Computation-Unit in Efficiency Graphs for DCOP Solvers , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[3]  Adam Jacoff,et al.  Traversability Metrics for Urban Search and Rescue Robots on Rough Terrain | NIST , 2007 .

[4]  Guillermo Rodríguez-Ortiz,et al.  A New Method to Evaluate Human-Robot System Performance , 2003, Auton. Robots.

[5]  Bartosz Mielczarek,et al.  Scenario-based performance analysis of routing protocols for mobile ad-hoc networks , 1999, MobiCom.

[6]  Joseph P. Macker,et al.  Mobile Ad hoc Networking (MANET): Routing Protocol Performance Issues and Evaluation Considerations , 1999, RFC.

[7]  Rajmohan Madhavan,et al.  Performance Analysis of Symbolic Road Recognition for On-Road Driving | NIST , 2007 .

[8]  Reid G. Smith,et al.  The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver , 1980, IEEE Transactions on Computers.

[9]  James S. Albus,et al.  Autonomy levels for unmanned systems (ALFUS) framework: an update , 2005, SPIE Defense + Commercial Sensing.

[10]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[11]  S.J.J. Smith,et al.  Empirical Methods for Artificial Intelligence , 1995 .

[12]  Thomas Fincannon,et al.  Task Performance Metrics in Human-Robot Interaction: Taking a Systems Approach , 2004 .

[13]  Andreas Haeberlen,et al.  Fallacies in Evaluating Decentralized Systems , 2006, IPTPS.

[14]  Makoto Yokoo,et al.  The Distributed Constraint Satisfaction Problem: Formalization and Algorithms , 1998, IEEE Trans. Knowl. Data Eng..

[15]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.

[16]  Christian Bessiere,et al.  Backtracking in Distributed Constraint Networks , 1998 .

[17]  H. Zimmermann,et al.  OSI Reference Model - The ISO Model of Architecture for Open Systems Interconnection , 1980, IEEE Transactions on Communications.

[18]  H. D. Rombach,et al.  The Goal Question Metric Approach , 1994 .

[19]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[20]  Jack J. Ference,et al.  Performance Evaluation of Integrated Vehicle-Based Safety Systems | NIST , 2007 .

[21]  Makoto Yokoo,et al.  An asynchronous complete method for distributed constraint optimization , 2003, AAMAS '03.

[22]  Lynne E. Parker,et al.  Fault-Tolerance Based Metrics for Evaluating System Performance in Multi-Robot Teams , 2006 .

[23]  Thomas E. Anderson,et al.  Quartz: a tool for tuning parallel program performance , 1990, SIGMETRICS '90.

[24]  B. Faltings,et al.  A Distributed, Complete Method for Multi-Agent Constraint Optimization , 2004 .

[25]  Felix A. Fischer,et al.  An integrated framework for adaptive reasoning about conversation patterns , 2005, AAMAS '05.

[26]  Stefano Carpin,et al.  USARSim: Providing a Framework for Multi-Robot Performance Evaluation | NIST , 2006 .

[27]  Victor R. Lesser,et al.  Solving distributed constraint optimization problems using cooperative mediation , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[28]  Rama Chellappa,et al.  An Information Theoretic Evaluation Criterion for 3D Reconstruction Algorithms , 2004 .

[29]  Ying Zhang,et al.  Parallel and distributed algorithms for finite constraint satisfaction problems , 1991, Proceedings of the Third IEEE Symposium on Parallel and Distributed Processing.

[30]  John Davin,et al.  Impact of problem centralization in distributed constraint optimization algorithms , 2005, AAMAS '05.

[31]  Stephen Balakirsky,et al.  Comparing Algorithms: Rules of Thumb and an Example , 2004 .

[32]  Katia P. Sycara,et al.  No-commitment branch and bound search for distributed constraint optimization , 2006, AAMAS '06.

[33]  Michael Wooldridge,et al.  Introduction to multiagent systems , 2001 .

[34]  Jean Scholtz,et al.  Development of an Evaluation Method for Acceptable Usability , 2007 .

[35]  D. Palmer,et al.  A Mixed Initiative Human-Robots Team Performance Assessment System for Use in Operational and Training Environments , 2004 .

[36]  Henrik I. Christensen,et al.  Results from a long-term study of a portable field robot in urban terrain , 2007, SPIE Defense + Commercial Sensing.

[37]  Barton P. Miller,et al.  Parallel program performance metrics: a comparison and validation , 1992, Proceedings Supercomputing '92.

[38]  L. C. Lee,et al.  The Stability, Scalability and Performance of Multi-agent Systems , 1998 .

[39]  Sesh Commuri,et al.  Evaluating Intelligence in Unmanned Ground Vehicle Teams , 2004 .

[40]  Henrik I. Christensen,et al.  Long-term study of a portable field robot in urban terrain , 2007, J. Field Robotics.