Making Models Match: Replicating an Agent-Based Model

Scientists have increasingly employed computer models in their work. Recent years have seen a proliferation of agent-based models in the natural and social sciences. But with the exception of a few "classic" models, most of these models have never been replicated by anyone but the original developer. As replication is a critical component of the scientific method and a core practice of scientists, we argue herein for an increased practice of replication in the agent-based modeling community, and for widespread discussion of the issues surrounding replication. We begin by clarifying the concept of replication as it applies to ABM. Furthermore we argue that replication may have even greater benefits when applied to computational models than when applied to physical experiments. Replication of computational models affects model verification and validation and fosters shared understanding about modeling decisions. To facilitate replication, we must create standards for both how to replicate models and how to evaluate the replication. In this paper, we present a case study of our own attempt to replicate a classic agent-based model. We begin by describing an agent-based model from political science that was developed by Axelrod and Hammond. We then detail our effort to replicate that model and the challenges that arose in recreating the model and in determining if the replication was successful. We conclude this paper by discussing issues for (1) researchers attempting to replicate models and (2) researchers developing models in order to facilitate the replication of their results.

[1]  Michael X Cohen,et al.  A Garbage Can Model of Organizational Choice. , 1972 .

[2]  Edward G. Carmines,et al.  Reliability and Validity Assessment , 1979 .

[3]  Seymour Papert,et al.  Mindstorms: Children, Computers, and Powerful Ideas , 1981 .

[4]  S. Shapin.  Laboratory life. The social construction of scientific facts , 1981, Medical History.

[5]  I. Janis.  Groupthink: Psychological Studies of Policy Decisions and Fiascoes , 1982 .

[6]  Susan Leigh Star,et al.  Changing Order: Replication and Induction in Scientific Practice by H. M. Collins (review) , 1988, Technology and Culture.

[7]  R. Axelrod.  An Evolutionary Approach to Norms , 1986, American Political Science Review.

[8]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[9]  P. Medawar,et al.  The Threat and the Glory: Reflections on Science and Scientists , 1990 .

[10]  W. Arthur.  Inductive Reasoning and Bounded Rationality , 1994 .

[11]  Robert L. Axtell,et al.  Aligning simulation models: A case study and results , 1996, Comput. Math. Organ. Theory.

[12]  C. Taylor.  Defining Science: A Rhetoric of Demarcation , 1996 .

[13]  R. Hegselmann,et al.  Simulating Social Phenomena , 1997 .

[14]  J. Forrester.  Industrial Dynamics , 1997 .

[15]  Robert Axelrod,et al.  Advancing the art of simulation in the social sciences , 1997, Complex..

[16]  R. Axelrod.  The Dissemination of Culture , 1997 .

[17]  George B. Kleindorfer,et al.  Validation in Simulation: Various Positions in the Philosophy of Science , 1998 .

[18]  David B. Fogel,et al.  Inductive reasoning and bounded rationality reconsidered , 1999, IEEE Trans. Evol. Comput..

[19]  Joshua M. Epstein,et al.  Agent-based computational models and generative social science , 1999, Complex..

[20]  Theodore C. Belding,et al.  Numerical Replication of Computer Simulations: Some Pitfalls and How To Avoid Them , 2000, ArXiv.

[21]  Capers Jones,et al.  Software Assessments, Benchmarks, and Best Practices , 2000 .

[22]  Scott Moss.  Canonical Tasks, Environments and Models for Social Simulation , 2000, Comput. Math. Organ. Theory.

[23]  Michael J. North,et al.  The Beer Dock : Three and a Half Implementations of the Beer Distribution Game , 2002 .

[24]  Joshua M. Epstein,et al.  Growing Artificial Societies: Social Science from the Bottom Up , 1996 .

[25]  Nicholas Mark Gotts,et al.  Comparative Analysis of Agent-based Social Simulations: GeoSim and FEARLUS models , 2003, J. Artif. Soc. Soc. Simul..

[26]  Ross A. Hammond,et al.  The Evolution of Ethnocentric Behavior , 2003 .

[27]  B. Edmonds,et al.  Replication, Replication and Replication: Some hard lessons from model alignment , 2003, J. Artif. Soc. Soc. Simul..

[28]  Juliette Rouchier.  Re-implementation of a Multi-agent Model aimed at Sustaining Experimental Economic Research: The case of simulations with emerging speculation , 2003, J. Artif. Soc. Soc. Simul..

[29]  Bruce Edmonds,et al.  Model-to-Model Analysis , 2003 .

[30]  Guillaume Deffuant,et al.  Comparing an Individual-based Model of Behaviour Diffusion with its Mean Field Aggregate Approximation , 2003, J. Artif. Soc. Soc. Simul..

[31]  Nicholas Mark Gotts,et al.  The Ghost in the Model (and Other Effects of Floating Point Arithmetic) , 2004, J. Artif. Soc. Soc. Simul..

[32]  José Manuel Galán,et al.  Appearances Can Be Deceiving: Lessons Learned Re-Implementing Axelrod's 'Evolutionary Approach to Norms' , 2005, J. Artif. Soc. Soc. Simul..

[33]  Uta Berger,et al.  Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology , 2005, Science.

[34]  J. Gareth Polhill,et al.  Lessons Learned from Converting the Artificial Stock Market to Interval Arithmetic , 2005, J. Artif. Soc. Soc. Simul..

[35]  B. Edmonds,et al.  Computational Simulation as Theoretical Experiment , 2005 .

[36]  J. Toner,et al.  Hydrodynamics and phases of flocks , 2005 .

[37]  Ross A. Hammond,et al.  The Evolution of Ethnocentrism , 2006 .

[38]  J. Giles.  The trouble with replication , 2006, Nature.

[39]  Nicholas Mark Gotts,et al.  What every agent-based modeller should know about floating point arithmetic , 2006, Environ. Model. Softw..

[40]  Birgit Müller,et al.  A standard protocol for describing individual-based and agent-based models , 2006 .

[41]  U. Wilensky,et al.  Thinking Like a Wolf, a Sheep, or a Firefly: Learning Biology Through Constructing and Testing Computational Theories—An Embodied Modeling Approach , 2006 .

[42]  J. Gareth Polhill,et al.  Is Your Model Susceptible to Floating-Point Errors? , 2006, J. Artif. Soc. Soc. Simul..

[43]  Ross A. Hammond,et al.  Evolution of contingent altruism when cooperation is expensive. , 2006, Theoretical population biology.

[44]  Sean Luke,et al.  Replication of Sugarscape Using MASON , 2007 .

[45]  Sean Luke,et al.  MASON : A Multi-Agent Simulation Environment , 2008 .