Contributions to Model Validation: Hierarchy, Process, and Cessation

In the domain of dynamic modeling and simulation, the assurance of model validity is a prominent challenge. An extensive number of contributions concerning model tests, terminology, and the epistemological foundations of validation have been elaborated. These contributions, however, do not fully answer the questions for novice modelers, namely, which validation tests to choose, when and how to apply them, and at what point to cease their formal validation efforts. Our intention here is to help close this gap by introducing a complexity hierarchy of validation tests, an integrative validation process, and a decision heuristic about when to stop formal validation efforts. The paper concludes by providing directions for future research.

[1]  Yaman Barlas,et al.  Model simplification and validation with indirect structure validity tests , 2006 .

[2]  Jay W. Forrester,et al.  “The” model versus a modeling “process” , 1985 .

[3]  John M. Wilson,et al.  Experiences in using a contingency factor-based validation methodology for spreadsheet DSS , 2008, J. Oper. Res. Soc..

[4]  K. Weick.  Theory Construction as Disciplined Imagination , 1989 .

[5]  Markus Schwaninger,et al.  System Dynamics as Model-Based Theory Building , 2008 .

[6]  Martin Schaffernicht.  Detecting and monitoring change in models , 2006 .

[7]  Geoff Coyle,et al.  The validation of commercial system dynamics models , 2000 .

[8]  George B. Kleindorfer,et al.  the Philosophy of Science and Validation in Simulation , 1993, Proceedings of 1993 Winter Simulation Conference - (WSC '93).

[9]  Muhittin Oral,et al.  The facets of the modeling and validation process in operations research , 1993 .

[10]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[11]  Silvia Ulli-Beer,et al.  How do we know who to include in collaborative research? Toward a method for the identification of experts , 2012, Eur. J. Oper. Res..

[12]  Paul N. Finlay,et al.  The Validation of Expert Systems — Contrasts with Traditional Methods , 1988 .

[13]  David C. Lane,et al.  The folding star: a comparative reframing and extension of validity concepts in system dynamics , 1995 .

[14]  Stefan N. Groesser,et al.  Mental models of dynamic systems: taking stock and looking ahead , 2012 .

[15]  Robert L. Eberlein,et al.  Reality check: A bridge between systems thinking and system dynamics , 1994 .

[16]  E F Wolstenholme,et al.  Qualitative vs quantitative modelling: the evolving balance , 1999, J. Oper. Res. Soc..

[17]  Y. Barlas.  Multiple tests for validation of system dynamics type of simulation models , 1989 .

[18]  Jack Homer,et al.  Partial-model testing as a validation tool for system dynamics (1983) , 2012 .

[19]  John D. Sterman,et al.  All models are wrong: reflections on becoming a systems scientist† , 2002 .

[20]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[21]  Paul N. Finlay,et al.  A survey of contingency factors affecting the validation of end-user spreadsheet-based decision support systems , 2000, J. Oper. Res. Soc..

[22]  Paul Bratley,et al.  A guide to simulation , 1983 .

[23]  J.P.C. Kleijnen.  Statistical validation of simulation models : A case study , 1995 .

[24]  Stefan N. Grösser.  Long-Term Innovation Diffusion in the Building Construction Industry: Empirically-Based Theory Building , 2008 .

[25]  A. Tversky,et al.  Prospect Theory : An Analysis of Decision under Risk Author ( s ) : , 2007 .

[26]  Rogelio Oliva,et al.  Structural dominance analysis and theory building in system dynamics , 2008 .

[27]  Osman Balci,et al.  A methodology for cost-risk analysis in the statistical validation of simulation models , 1981, CACM.

[28]  Stefan N. Groesser,et al.  A comprehensive method for comparing mental models of dynamic systems , 2011, Eur. J. Oper. Res..

[29]  S. Bruppacher,et al.  Decisions in the Construction Planning Process : Development of a Dynamic Model about Individual ’ s Energy Efficiency Intention over Time , 2007 .

[30]  J. Forrester.  Industrial Dynamics , 1997 .

[31]  Stefan N. Grösser,et al.  Diffusion dynamics of energy-efficient innovations in the residential building environment , 2006 .

[32]  Jay W. Forrester,et al.  System dynamics—the next fifty years , 2007 .

[33]  John M. Wilson,et al.  Validation of decision-aiding spreadsheets: the influence of contingency factors , 2004, J. Oper. Res. Soc..

[34]  Yaman Barlas,et al.  Formal aspects of model validity and validation in system dynamics , 1996 .

[35]  Roger I. Hall,et al.  A System Pathology of an Organization: The Rise and Fall of the Old Saturday Evening Post. , 1976 .

[36]  A. Strauss,et al.  Grounded Theory in Practice , 1997 .

[37]  Paul N. Finlay,et al.  Validity of Decision Support Systems: Towards a Validation Methodology , 1997 .

[38]  Robert G. Sargent,et al.  Validation and verification of simulation models , 1999, Proceedings of the 2004 Winter Simulation Conference, 2004..

[39]  Robert G. Sargent,et al.  Verification and validation of simulation models , 2009, IEEE Engineering Management Review.

[40]  Jack Homer,et al.  Why we iterate: scientific modeling in theory and practice , 1996 .

[41]  Yaman Barlas,et al.  Automated parameter specification in dynamic feedback models based on behavior pattern features , 2011 .

[42]  Saul I. Gass,et al.  Feature Article - Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis , 1983, Oper. Res..

[43]  Osman Balci.  Validation, verification, and testing techniques throughout the life cycle of a simulation study , 1994, WSC '94.

[44]  David N. Ford.  A behavioral approach to feedback loop dominance analysis , 1999 .

[45]  John D. Sterman,et al.  Diagnosing surprise model behavior: A tool for evolving behavioral and policy insights (1981) , 1991 .

[46]  B. Glaser.  Theoretical Sensitivity: Advances in the Methodology of Grounded Theory , 1978 .

[47]  Markus Schwaninger,et al.  System Dynamics Modeling: Validation for Quality Assurance , 2009, Encyclopedia of Complexity and Systems Science.

[48]  Yaman Barlas,et al.  An autocorrelation function test for out put validation , 1990, Simul..

[49]  Jack P. C. Kleijnen,et al.  Statistical validation of simulation models , 1995 .

[50]  Maurice Landry,et al.  Revisiting the issue of model validation in OR: An epistemological view , 1993 .

[51]  Rogelio Oliva,et al.  Model calibration as a testing strategy for system dynamics models , 2003, Eur. J. Oper. Res..

[52]  Jay W. Forrester,et al.  Policies, decisions and information sources for modeling☆ , 1992 .

[53]  Burak Güneralp.  Towards coherent loop dominance analysis: progress in eigenvalue elasticity analysis , 2006 .

[54]  Yaman Barlas,et al.  Philosophical roots of model validation: Two paradigms , 1990 .

[55]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .