An Objective-Based Perspective on Assessment of Model-Supported Policy Processes

Simulation models, being in use for a long time in natural sciences and engineering domains, are diffusing to a wider context including policy analysis studies. The differences between the nature of the domain of application, as well as the increased variety of usage partially induced by this difference naturally imply new challenges to be overcome. One of these challenges is related to the assessment of the simulation-based outcomes in terms of their reliability and relevance in the policy context being studied. The importance of this assessment is twofold. First of all, it is all about conducting a high quality policy study with effective results. However, the quality of the study does not necessarily imply acceptance of the results by the clients and/or colleagues. This problem of policy analysts increases the importance of such an assessment; an effective assessment may induce the acceptance of the conclusions drawn from the study by the clients and/or colleagues. The main objective of this paper is to introduce an objective-based assessment perspective for simulation model-supported policy studies. As a first step towards such a goal, an objective-based classification of models is introduced. Based on that, we will discuss the importance of different aspects of the assessment for each type. In doing so, we aim to provide a structured discussion that may serve as a sort of methodological guideline to be used by policy analysts, and also by clients.

[1]  B. Soares-Filho,et al.  Modelling conservation in the Amazon basin , 2006, Nature.

[2]  Ulrich Frank,et al.  Epistemological Perspectives on Simulation , 2005 .

[3]  Paul Windrum,et al.  Empirical Validation of Agent-Based Models: Alternatives and Prospects , 2007, J. Artif. Soc. Soc. Simul..

[4]  Alex Schmid.  What is the Truth of Simulation? , 2005, J. Artif. Soc. Soc. Simul..

[5]  L. Argote.  Organizational Learning: Creating, Retaining and Transferring Knowledge , 1999 .

[6]  M. Stadler,et al.  Policy strategies and paths to promote sustainable energy systems--The dynamic Invert simulation tool , 2007 .

[7]  Luis F. Luna-Reyes,et al.  Collecting and analyzing qualitative data for system dynamics: methods and models , 2003 .

[8]  Evrim Dalkõran.  SCUBA DIVING SIMULATOR: TESTING REAL TIME DECISION MAKING IN A FEEDBACK ENVIRONMENT , 2006 .

[9]  François Bousquet,et al.  Companion Modeling, Conflict Resolution, and Institution Building: Sharing Irrigation Water in the Lingmuteychu Watershed, Bhutan , 2006 .

[10]  Björn Niehaves,et al.  A Framework for Epistemological Perspectives on Simulation , 2005, J. Artif. Soc. Soc. Simul..

[11]  L. Argote,et al.  Learning Curves in Manufacturing , 1990, Science.

[12]  Phillip D. Stroud,et al.  Spatial Dynamics of Pandemic Influenza in a Massive Artificial Society , 2007, J. Artif. Soc. Soc. Simul..

[13]  George P. Richardson,et al.  Group model building: adding more science to the craft , 1997 .

[14]  Ph. Le Grusse,et al.  Use of a decision support system and a simulation game to help collective decision-making in water management , 2008 .

[15]  L. H. Immers,et al.  Gaming Approach Route 26: a combination of computer simulation, design tools and social interaction , 2002, J. Artif. Soc. Soc. Simul..

[16]  Jay Wright Forrester,et al.  Urban Dynamics , 1969 .

[17]  Thomas Brenner,et al.  Policy Advice Derived from Simulation Models , 2009, J. Artif. Soc. Soc. Simul..

[18]  Aldo A. Zagonel,et al.  MODEL CONCEPTUALIZATION IN GROUP MODEL BUILDING : A REVIEW OF THE LITERATURE EXPLORING THE TENSION BETWEEN REPRESENTING REALITY AND NEGOTIATING A SOCIAL ORDER , 2002 .

[19]  I. Mayer,et al.  Perspectives on Policy Analysis: A Framework for Understanding and Design , 2004 .

[20]  Levent Yilmaz,et al.  Validation and verification of social processes within agent-based computational organization models , 2006, Comput. Math. Organ. Theory.

[21]  Osman Balci.  Validation, verification, and testing techniques throughout the life cycle of a simulation study , 1994, WSC '94.

[22]  Joseph P. Martino.  Handbook of systems analysis: Craft issues and procedural choices , 1990 .

[23]  Keiki Takadama,et al.  Micro- and Macro-Level Validation in Agent-Based Simulation: Reproduction of Human-Like Behaviors and Thinking in a Sequential Bargaining Game , 2008, J. Artif. Soc. Soc. Simul..

[24]  Etiënne A. J. A. Rouwette,et al.  Group model building as mutual persuasion , 2003 .

[25]  Ana Maria Ramanath,et al.  The Design of Participatory Agent-Based Social Simulations , 2004, J. Artif. Soc. Soc. Simul..

[26]  Ewart R. Carson,et al.  Model validation: philosophy, methodology and examples , 1990 .

[27]  R G Coyle,et al.  System Dynamics Modelling: A Practical Approach , 1996 .

[28]  Jeroen Struben,et al.  Essays on transition challenges for alternative propulsion vehicles and transportation systems , 2006 .

[29]  Hui Zhao.  Simulation and analysis of dealers' returns distribution strategy , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[30]  Flaminio Squazzoni,et al.  Does Empirical Embeddedness Matter? Methodological Issues on Agent-Based Models for Analytical Social Science , 2005, J. Artif. Soc. Soc. Simul..

[31]  Scott Moss,et al.  Alternative Approaches to the Empirical Validation of Agent-Based Models , 2007, J. Artif. Soc. Soc. Simul..

[32]  Warren E. Walker,et al.  FORWARD--FREIGHT OPTIONS FOR ROAD, WATER, AND RAIL FOR THE DUTCH. FINAL REPORT , 1996 .

[33]  François Bousquet,et al.  Using Self-Designed Role-Playing Games and a Multi-Agent System to Empower a Local Decision-Making Process for Land Use Management: The SelfCormas Experiment in Senegal , 2003, J. Artif. Soc. Soc. Simul..

[34]  Thomas Brenner,et al.  A Taxonomy of Inference in Simulation Models , 2007 .

[35]  Jan Rotmans,et al.  Image: An Integrated Model to Assess the Greenhouse Effect , 1990 .

[36]  Robert G. Sargent,et al.  Validation and verification of simulation models , 1999, Proceedings of the 2004 Winter Simulation Conference, 2004..

[37]  K. G. Troitzsch.  VALIDATING SIMULATION MODELS , 2004 .

[38]  K. Stave.  Using system dynamics to improve public participation in environmental decisions , 2002 .

[39]  Robert G. Sargent,et al.  Validating simulation models , 1983, WSC '83.

[40]  B. Edmonds,et al.  Sociology and Simulation: Statistical and Qualitative Cross‐Validation1 , 2005, American Journal of Sociology.

[41]  D. Cummings,et al.  Strategies for mitigating an influenza pandemic , 2006, Nature.

[42]  François Bousquet,et al.  Role-playing games for opening the black box of multi-agent systems: method and lessons of its application to Senegal River Valley irrigated systems , 2001, J. Artif. Soc. Soc. Simul..

[43]  D. Menniti,et al.  Simulation of producers behaviour in the electricity market by evolutionary games , 2008 .

[44]  William N. Dunn,et al.  A pragmatic strategy for discovering and testing threats to the validity of sociotechnical experiments , 2002, Simul. Model. Pract. Theory.

[45]  J. Scholes,et al.  Soft Systems Methodology in Action: Including a 30-Year Retrospective , 2000 .

[46]  Johannes Lenhard,et al.  Validation of Simulation: Patterns in the Social and Natural Sciences , 2005, J. Artif. Soc. Soc. Simul..

[47]  Pieter W. G. Bots,et al.  Participatory Model Construction and Model Use in Natural Resource Management: a Framework for Reflection , 2008 .

[48]  G. Nigel Gilbert,et al.  Caffè Nero: the Evaluation of Social Simulation , 2005, J. Artif. Soc. Soc. Simul..

[49]  G. Nigel Gilbert,et al.  Simulation for the social scientist , 1999 .

[50]  Thomas C. Schelling,et al.  Dynamic models of segregation , 1971 .

[51]  B. Hobbs,et al.  Using game theory to analyze electric transmission pricing policies in the United States , 1992 .

[52]  Yaman Barlas,et al.  Formal aspects of model validity and validation in system dynamics , 1996 .

[53]  Steffen Bayer,et al.  Business dynamics: Systems thinking and modeling for a complex world , 2004 .

[54]  N. Lucas,et al.  Competition in the UK market for electricity generating capacity: A game theory analysis , 1993 .

[55]  Y. Barlas,et al.  Environmental sustainability in an agricultural development project: a system dynamics approach. , 2002, Journal of environmental management.

[56]  Klaus Hubacek,et al.  ‘Who’s in the Network?’ When Stakeholders Influence Data Analysis , 2008 .

[57]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[58]  G. Rowe,et al.  Public Participation Methods: A Framework for Evaluation , 2000 .

[59]  Olivier Barreteau,et al.  Our Companion Modelling Approach , 2003, J. Artif. Soc. Soc. Simul..