Validating agent based social systems models

Validating social systems is not a trivial task. The paper outlines some of our past efforts in validating models of social systems with cognitively detailed agents. It also presents some of the challenges faced by us. A social system built primarily of cognitively detailed agents can provide multiple levels of correspondence, both at observable and abstract aggregated levels. Such a system can also pose several challenges including large feature spaces, issues in information elicitation with database, experts and news feeds, counterfactuals, fragmented theoretical base, and limited funding for validation. Our own approach to validity assessment is to consider the entire life cycle and assess the validity under four broad dimensions of methodological validity, internal validity, external validity and qualitative, causal and narrative validity. In the past, we have employed a triangulation of multiple validation techniques, including face validation as well as formal validation tests including correspondence testing.

[1]  Roberto Leombruni,et al.  A Common Protocol for Agent-Based Social Simulation , 2006, J. Artif. Soc. Soc. Simul..

[2]  A. H. Murphy,et al.  What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting , 1993 .

[3]  G. Nigel Gilbert,et al.  Open Problems in Using Agent-Based Models in Industrial and Labor Dynamics , 2004, Adv. Complex Syst..

[4]  Darren Schreiber.  Validating Agent –Based Models: From Metaphysics to Applications , 2003 .

[5]  Barry G. Silverman,et al.  Gaming and Simulating Ethno-Political Conflicts , 2009 .

[6]  Nicholas Griffin,et al.  THE PRINCIPLES OF SCIENTIFIC THINKING , 1971 .

[7]  B. Edmonds,et al.  Sociology and Simulation: Statistical and Qualitative Cross‐Validation1 , 2005, American Journal of Sociology.

[8]  Gnana Bharathy,et al.  Sociocultural Games for Training and Analysis , 2007, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[9]  Jaime Simão Sichman,et al.  Multi-Agent-Based Simulation , 2002, Lecture Notes in Computer Science.

[10]  Sander van der Hoog,et al.  On Multi-Agent Based Simulation , 2004 .

[11]  Scott E. Page,et al.  Computational and Mathematical Modeling in the Social Sciences , 2005 .

[12]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[13]  Claudia Pahl-Wostl,et al.  The Dynamic Nature of Ecosystems: Chaos and Order Entwined , 1995 .

[14]  Gnana Bharathy,et al.  Modeling factions for ‘effects based operations’, part II: behavioral game theory , 2008, Comput. Math. Organ. Theory.

[15]  John F. Agoglia,et al.  Measuring Progress in Conflict Environments (MPICE): A Metrics Framework , 2008 .

[16]  Robert L. Axtell,et al.  Aligning simulation models: A case study and results , 1996, Comput. Math. Organ. Theory.

[17]  Lawrence Z. Markosian,et al.  Verification and Validation , 2011 .

[18]  Aristides Dasso.  Verification, Validation And Testing In Software Engineering , 2006 .

[19]  Scott de Marchi.  Computational and Mathematical Modeling in the Social Sciences , 2005 .

[20]  A. Tversky,et al.  Choices, Values, and Frames , 2000 .

[21]  Paul Windrum,et al.  Empirical Validation of Agent Based Models: A Critical Survey , 2006 .

[22]  Booncharoen Sirinaovakul,et al.  Introduction to the Special Issue , 2002, Comput. Intell..

[23]  Barry G. Silverman,et al.  Systems Social Seience: A design inquiry approach for stabilization and reconstruction of social systems , 2010, Intell. Decis. Technol..

[24]  Dean S. Hartley.  Verification & validation in military simulations , 1997, WSC '97.

[25]  Cyril Goutte,et al.  Note on Free Lunches and Cross-Validation , 1997, Neural Computation.

[26]  Osman Balci,et al.  Verification, Validation, and Testing , 2007 .

[27]  Bruce Edmonds,et al.  Model-to-Model Analysis , 2003 .

[28]  Gnana Bharathy,et al.  Modeling factions for “effects based operations”: part I—leaders and followers , 2007, Comput. Math. Organ. Theory.

[29]  P. Tetlock.  Expert Political Judgment: How Good Is It? How Can We Know? , 2005 .

[30]  Bruce Edmonds.  When Simple Measures Fail: Characterising Social Networks Using Simulation , 2005 .

[31]  B. Silverman,et al.  Modeling the Personality & Cognition of Leaders , 2005 .

[32]  Olivier Barreteau,et al.  Our Companion Modelling Approach , 2003, J. Artif. Soc. Soc. Simul..

[33]  John A. Sokolowski,et al.  Modeling and Simulation Fundamentals: Theoretical Underpinnings and Practical Domains , 2010 .

[34]  Bruce Edmonds,et al.  From KISS to KIDS - An 'Anti-simplistic' Modelling Approach , 2004, MABS.

[35]  Kevin A. Gluck,et al.  Introduction to the Special Issue , 2008, Cogn. Sci..

[36]  David F. Midgley,et al.  Building and assurance of agent-based models: An example and challenge to the field , 2007 .

[37]  Michael J. Dziedzic,et al.  The quest for viable peace : international intervention and strategies for conflict transformation , 2005 .