A generic testing framework for agent-based simulation models

Agent-based modelling and simulation (ABMS) had an increasing attention during the last decade. However, the weak validation and verification of agent-based simulation models makes ABMS hard to trust. There is no comprehensive tool set for verification and validation of agent-based simulation models, which demonstrates that inaccuracies exist and/or reveals the existing errors in the model. Moreover, on the practical side, many ABMS frameworks are in use. In this sense, we designed and developed a generic testing framework for agent-based simulation models to conduct validation and verification of models. This paper presents our testing framework in detail and demonstrates its effectiveness by showing its applicability on a realistic agent-based simulation case study.

[1]  Michael Pidd,et al.  Computer Simulation in Management Science (3rd Edition) , 1998 .

[2]  Eric Solano,et al.  Agent-Based Model (ABM) Validation Considerations , 2011 .

[3]  Franziska Klügl-Frohnmeyer,et al.  A validation methodology for agent-based simulations , 2008, SAC.

[4]  Franziska Klügl-Frohnmeyer,et al.  About the Role of the Environment in Multi-agent Simulations , 2004, E4MAS.

[5]  Giorgio Fagiolo,et al.  An Evolutionary Model of Endogenous Business Cycles , 2006 .

[6]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[7]  T. S. Miles,et al.  Threshold depolarization measurements in resting human motoneurones , 1991, Journal of Neuroscience Methods.

[8]  Michael C. Feathers.  Working Effectively with Legacy Code , 2004, XP/Agile Universe.

[9]  Claudia Raibulet,et al.  Towards Evaluation Mechanisms for Runtime Adaptivity: From Case Studies to Metrics , 2009, 2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns.

[10]  Patrick Taillandier,et al.  GAMA: A Simulation Platform That Integrates Geographical Information Data, Agent-Based Modeling and Multi-scale Control , 2010, PRIMA.

[11]  Susan Stepney,et al.  Reflections on the Simulation of Complex Systems for Science , 2010, 2010 15th IEEE International Conference on Engineering of Complex Computer Systems.

[12]  Gauthier Picard,et al.  Characterizing and Evaluating Problem Solving Self-* Systems , 2009, 2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns.

[13]  Uirá Kulesza,et al.  Unit testing in multi-agent systems using mock agents and aspects , 2006, SELMAS '06.

[14]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[15]  W H Calvin,et al.  Steps in production of motoneuron spikes during rhythmic firing. , 1972, Journal of neurophysiology.

[16]  John Thangarajah,et al.  Testing in Multi-Agent Systems , 2009, AOSE.

[17]  Craig Larman,et al.  Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development (3rd Edition) , 1997 .

[18]  Thomas Brenner,et al.  Empirical Calibration of Simulation Models , 2004 .

[19]  Uta Berger,et al.  Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology , 2005, Science.

[20]  Franziska Kl.  A Validation Methodology for Agent-Based Simulations , 2008 .

[21]  Juan A. Botía Blaya,et al.  Infrastructure for Forensic Analysis of Multi-Agent Systems , 2009, ProMAS.

[22]  Paul Windrum,et al.  Empirical Validation of Agent-Based Models: Alternatives and Prospects , 2007, J. Artif. Soc. Soc. Simul..

[23]  Osman Balci.  Validation, verification, and testing techniques throughout the life cycle of a simulation study , 1994, WSC '94.

[24]  Stewart Robinson,et al.  Simulation model verification and validation: increasing the users' confidence , 1997, WSC '97.

[25]  Cristina Montañola-Sales,et al.  Agent-based simulation validation: A case study in demographic simulation , 2011 .

[26]  Adelinde M. Uhrmacher,et al.  Agent-oriented simulation , 2003 .

[27]  Ilene Burnstein,et al.  Practical Software Testing , 2003, Springer Professional Computing.

[28]  Michael Lees,et al.  Validation of Agent-Based Simulation through Human Computation: An Example of Crowd Simulation , 2011, MABS.

[29]  Fabien Michel,et al.  Modeling dynamic environments in multi-agent simulation , 2007, Autonomous Agents and Multi-Agent Systems.

[30]  Joshua M. Epstein,et al.  Agent-based computational models and generative social science , 1999, Complex..

[31]  Michael J. North,et al.  Experiences creating three implementations of the repast agent modeling toolkit , 2006, TOMC.

[32]  Juan Pavón,et al.  Agent-Based Simulation Replication: A Model Driven Architecture Approach , 2005, MICAI.

[33]  Osman Balci.  Principles and techniques of simulation validation, verification, and testing , 1995, WSC '95.

[34]  Jorge J. Gómez-Sanz,et al.  Intelligent data analysis applied to debug complex software systems , 2009, Neurocomputing.

[35]  Tom De Wolf,et al.  Engineering self-organising emergent systems with simulation-based scientific analysis , 2005 .

[36]  HARDSCAPE proaucis,et al.  Tools of the trade , 1995, Nature.

[37]  P. Schwindt,et al.  Factors influencing motoneuron rhythmic firing: results from a voltage-clamp study. , 1982, Journal of neurophysiology.

[38]  Steven C Bankes,et al.  Agent-based modeling: A revolution? , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[39]  Oguz Dikenelli,et al.  Agent-based Exploration of Wirings of Biological Neural Networks: Position Paper , 2012, ArXiv.

[40]  William Rand,et al.  Making Models Match: Replicating an Agent-Based Model , 2007, J. Artif. Soc. Soc. Simul..

[41]  R. Powers,et al.  Black box revisited: a technique for estimating postsynaptic potentials in neurons , 2005, Trends in Neurosciences.

[42]  Luis Antunes,et al.  Multi-Agent-Based Simulation VII, International Workshop, MABS 2006, Hakodate, Japan, May 8, 2006, Revised and Invited Papers , 2007, MABS.

[43]  Franziska Klügl-Frohnmeyer,et al.  Multiagent Simulation Model Design Strategies , 2009, MALLOW.

[44]  Klaus G. Troitzsch.  Multilevel Simulation , 1995, Social Science Microsimulation.

[45]  D. Capera,et al.  The AMAS theory for complex problem solving based on self-organizing cooperative agents , 2003, WET ICE 2003. Proceedings. Twelfth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2003..

[46]  Steven F. Railsback,et al.  Agent-Based and Individual-Based Modeling: A Practical Introduction , 2011 .

[47]  Takao Terano.  Exploring the Vast Parameter Space of Multi-Agent Based Simulation , 2006, MABS.

[48]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[49]  Oguz Dikenelli,et al.  Simulating Human Single Motor Units Using Self-Organizing Agents , 2012, 2012 IEEE Sixth International Conference on Self-Adaptive and Self-Organizing Systems.

[50]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[51]  Sumit Ghosh.  On the concept of dynamic multi-level simulation , 1986, ANSS '86.

[52]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[53]  Gregory R. Madey,et al.  Tools of the Trade: A Survey of Various Agent Based Modeling Platforms , 2009, J. Artif. Soc. Soc. Simul..

[54]  Sean Luke,et al.  MASON: A Multiagent Simulation Environment , 2005, Simul..

[55]  Elizabeth Sklar,et al.  NetLogo, a Multi-agent Simulation Environment , 2007, Artificial Life.