A content analysis-based approach to explore simulation verification and identify its current challenges

Verification is a crucial process to facilitate the identification and removal of errors within simulations. This study explores semantic changes to the concept of simulation verification over the past six decades using a data-supported, automated content analysis approach. We collect and utilize a corpus of 4,047 peer-reviewed Modeling and Simulation (M&S) publications dealing with a wide range of studies of simulation verification from 1963 to 2015. We group the selected papers by decade of publication to provide insights and explore the corpus from four perspectives: (i) the positioning of prominent concepts across the corpus as a whole; (ii) a comparison of the prominence of verification, validation, and Verification and Validation (V&V) as separate concepts; (iii) the positioning of the concepts specifically associated with verification; and (iv) an evaluation of verification’s defining characteristics within each decade. Our analysis reveals unique characterizations of verification in each decade. The insights gathered helped to identify and discuss three categories of verification challenges as avenues of future research, awareness, and understanding for researchers, students, and practitioners. These categories include conveying confidence and maintaining ease of use; techniques’ coverage abilities for handling increasing simulation complexities; and new ways to provide error feedback to model users.

[1]  Mark J. Damborg,et al.  Credibility of models (panel discussion) , 1985 .

[2]  Yaman Barlas,et al.  Sensitivity analysis for models with multiple behavior modes: a method based on behavior pattern measures , 2016 .

[3]  Loo Hay Lee,et al.  Stochastically Constrained Ranking and Selection via SCORE , 2014, ACM Trans. Model. Comput. Simul..

[4]  Osman Balci Credibility assessment of simulation results , 1986, WSC '86.

[5]  K. Keller,et al.  Model-based Approach to Validation and Verification of Flight Critical Software , 2008, 2008 IEEE Aerospace Conference.

[6]  Yong He,et al.  BrainNet Viewer: A Network Visualization Tool for Human Brain Connectomics , 2013, PloS one.

[7]  Martin Kunc,et al.  Hybrid simulation modelling in operational research: A state-of-the-art review , 2019, Eur. J. Oper. Res..

[8]  Bin Chen,et al.  Modeling Spatial Contacts for Epidemic Prediction in a Large-Scale Artificial City , 2016, J. Artif. Soc. Soc. Simul..

[9]  R. Nance,et al.  Independent verification and validation: a missing link in simulation methodology? , 1996, Proceedings Winter Simulation Conference.

[10]  Christopher J. Lynch,et al.  Observations on the practice and profession of modeling and simulation: A survey approach , 2018, Simul..

[11]  Wei-Tek Tsai,et al.  Multi-tenant Verification-as-a-Service (VaaS) in a cloud , 2016, Simul. Model. Pract. Theory.

[12]  Christopher J. Lynch,et al.  Applying statistical debugging for enhanced trace validation of agent-based models , 2017, Simul..

[13]  Christopher J. Lynch,et al.  Towards airspace rules for future UAS-based delivery , 2016, 2016 Winter Simulation Conference (WSC).

[14]  D. Higdon,et al.  Accelerating Markov Chain Monte Carlo Simulation by Differential Evolution with Self-Adaptive Randomized Subspace Sampling , 2009 .

[15]  Thomas Sauerbier UMDBS - A New Tool for Dynamic Microsimulation , 2002, J. Artif. Soc. Soc. Simul..

[16]  Adelinde M. Uhrmacher,et al.  Automatic Runtime Adaptation for Component-Based Simulation Algorithms , 2015, TOMC.

[17]  James D. Arthur,et al.  Independent verification and validation: a missing link in simulation methodology? , 1996, Winter Simulation Conference.

[18]  R. B. Whitner,et al.  Guidelines for selecting and using simulation model verification techniques , 1989, WSC '89.

[19]  Jose J. Padilla,et al.  Fine-Scale Prediction of People's Home Location Using Social Media Footprints , 2018, SBP-BRiMS.

[20]  Juho Rousu,et al.  Comparative Genome-Scale Reconstruction of Gapless Metabolic Networks for Present and Ancestral Species , 2014, PLoS Comput. Biol..

[21]  Andreas Tolk,et al.  Standards for Distributed Simulation , 2012 .

[22]  Barry L. Nelson,et al.  ‘Some tactical problems in digital simulation’ for the next 10 years , 2016, J. Simulation.

[23]  R. Sargent,et al.  Some subjective validation methods using graphical displays of data , 1996, Proceedings Winter Simulation Conference.

[24]  Semih Bilgen,et al.  UML-Based Conceptual Models and V&V , 2010 .

[25]  Robert S. Hotchkiss,et al.  The auditorialization of scientific information , 1991, Proceedings of the 1991 ACM/IEEE Conference on Supercomputing (Supercomputing '91).

[26]  Valeriy Vyatkin,et al.  Requirement verification in simulation-based automation testing , 2016, 2016 IEEE 14th International Conference on Industrial Informatics (INDIN).

[27]  Hui Xu,et al.  Automated numerical simulation of biological pattern formation based on visual feedback simulation framework , 2017, PloS one.

[28]  D. Post,et al.  Computational Science Demands a New Paradigm , 2005 .

[29]  Ross Gore,et al.  Forecasting Changes in Religiosity and Existential Security with an Agent-Based Model , 2018, J. Artif. Soc. Soc. Simul..

[30]  Umut Durak,et al.  Special issue on modeling and simulation in the era of big data and cloud computing: theory, framework and tools , 2017, Simul..

[31]  D. Pace Modeling and simulation verification and validation challenges , 2004 .

[32]  R. Henneman Vision and audition as sensory channels for communication , 1952 .

[33]  Christopher J. Lynch,et al.  A taxonomy for classifying terminologies that describe simulations with multiple models , 2015, 2015 Winter Simulation Conference (WSC).

[34]  Christopher J. Lynch,et al.  Identifying Unexpected Behaviors of Agent-Based Models Through Spatial Plots and Heat Maps , 2019, Understanding Complex Systems.

[35]  Tsong Yueh Chen,et al.  A preliminary survey on software testing practices in Australia , 2004, 2004 Australian Software Engineering Conference. Proceedings..

[36]  Christopher J. Lynch,et al.  Modeling and simulation as a service (MSaaS) for education: learning STEM concepts through simulation use and building , 2015, SummerSim.

[37]  Osman Balci Golden Rules of Verification, Validation, Testing, and Certification of Modeling and Simulation Applications , 2010 .

[38]  M. Johnson,et al.  Circulating microRNAs in Sera Correlate with Soluble Biomarkers of Immune Activation but Do Not Predict Mortality in ART Treated Individuals with HIV-1 Infection: A Case Control Study , 2015, PloS one.

[39]  Christopher J. Lynch,et al.  A multi-paradigm modeling framework for modeling and simulating problem situations , 2014, Proceedings of the Winter Simulation Conference 2014.

[40]  J. M. Voogd,et al.  The Generic Methodology for Verification and Validation to support acceptance of models, simulations and data , 2013 .

[41]  Forrest Stonedahl,et al.  The Complexities of Agent-Based Modeling Output Analysis , 2015, J. Artif. Soc. Soc. Simul..

[42]  Joshua M. Epstein,et al.  Growing Artificial Societies: Social Science from the Bottom Up , 1996 .

[43]  M. Jacomy,et al.  ForceAtlas2, a Continuous Graph Layout Algorithm for Handy Network Visualization Designed for the Gephi Software , 2014, PloS one.

[44]  Richard Torkar,et al.  A survey on testing and reuse , 2003, Proceedings 2003 Symposium on Security and Privacy.

[45]  Christopher J. Lynch,et al.  Formal methods, statistical debugging and exploratory analysis in support of system development: Towards a verification and validation calculator tool , 2016, Int. J. Model. Simul. Sci. Comput..

[46]  Scott Fortmann-Roe,et al.  Insight Maker: A general-purpose tool for web-based modeling & simulation , 2014, Simul. Model. Pract. Theory.

[47]  William W. Gaver The SonicFinder: An Interface That Uses Auditory Icons , 1989, Hum. Comput. Interact..

[48]  Christopher J. Lynch,et al.  Emergent behavior identification within an agent-based model of the Ballistic Missile Defense System using statistical debugging , 2016 .

[49]  Gennady L. Andrienko,et al.  Exploratory analysis of spatial and temporal data - a systematic approach , 2005 .

[50]  Barry G. Silverman,et al.  What is a good pattern of life model? Guidance for simulations , 2019, Simul..

[51]  Ernest J. McCormick,et al.  Human beings in relation to equipment. , 1957 .

[52]  Vladimír Bureš,et al.  Experimental Evaluation of Suitability of Selected Multi-Criteria Decision-Making Methods for Large-Scale Agent-Based Simulations , 2016, PloS one.

[53]  Christopher J. Lynch,et al.  Cloud-based simulators: Making simulations accessible to non-experts and experts alike , 2014, Proceedings of the Winter Simulation Conference 2014.

[54]  Heidrun Schumann,et al.  Visualizing time-oriented data - A systematic view , 2007, Comput. Graph..

[55]  B. H. McCormick,et al.  Visualization in scientific computing , 1995 .

[56]  Frederic D. McKenzie,et al.  Visualization and rule validation in human-behavior representation , 2008 .

[57]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[58]  Colin Seymour-Ure,et al.  Content Analysis in Communication Research. , 1972 .

[59]  J. Mezrich,et al.  Dynamic Representation of Multivariate Time Series Data , 1984 .

[60]  Boxin Tang,et al.  Latin Hypercubes and Space-Filling Designs , 2015, 2203.06334.

[61]  Joshua M. Epstein,et al.  Understanding Anasazi culture change through agent-based modeling , 2000 .

[62]  Robert G. Sargent,et al.  An overview of verification and validation of simulation models , 1987, WSC '87.

[63]  Michael J. Seiler,et al.  Applying Latin hypercube sampling to agent-based models: Understanding foreclosure contagion effects , 2013 .

[64]  Derek Bingham,et al.  Handbook of Design and Analysis of Experiments , 2015 .

[65]  Frederick F. Stephan,et al.  Descriptive and Sampling Statistics. , 1948 .

[66]  Edward Rolf Tufte,et al.  The visual display of quantitative information , 1985 .

[67]  R. Weber Basic Content Analysis , 1986 .

[68]  James P. Ahrens,et al.  Verifying Scientific Simulations via Comparative and Quantitative Visualization , 2010, IEEE Computer Graphics and Applications.

[69]  Joshua M. Epstein,et al.  Agent_Zero: Toward Neurocognitive Foundations for Generative Social Science , 2014 .

[70]  Kwan-Liu Ma,et al.  Contact Trees: Network Visualization beyond Nodes and Edges , 2016, PloS one.

[71]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .

[72]  R. D. Hurrion An interactive visual simulation system for industrial management , 1980 .

[73]  Bernard P. Zeigler,et al.  Towards a framework for more robust validation and verification of simulation models for systems of systems , 2016 .

[74]  I. Notz William Expected Improvement Designs , 2015 .

[75]  Justin E. Lane Method, Theory, and Multi-Agent Artificial Intelligence: Creating computer models of complex social interaction , 2014 .

[76]  Giorgio Fagiolo,et al.  Spatial Interactions in Dynamic Decentralised Economies: a Review , 1998 .

[77]  Alexander Lex,et al.  Ten simple rules to create biological network figures for communication , 2019, PLoS Comput. Biol..

[78]  Andreas Tolk,et al.  Engineering Principles of Combat Modeling and Distributed Simulation: Tolk/Engineering Principles Combat Modeling , 2012 .

[79]  Paul A. Strooper,et al.  A state-of-practice questionnaire on verification and validation for concurrent programs , 2006, PADTAD '06.

[80]  Ruth Reed,et al.  Descriptive and Sampling Statistics , 1947 .

[81]  J. Ahmad,et al.  Petri Net and Probabilistic Model Checking Based Approach for the Modelling, Simulation and Verification of Internet Worm Propagation , 2015, PloS one.

[82]  Hassan Rajaei,et al.  A dististributed simulation platform for cloud computing , 2017, IEEE CNS 2017.

[83]  Ray J. Paul,et al.  Web-based simulation: revolution or evolution? , 2000, TOMC.

[84]  J. Forrester Industrial Dynamics , 1997 .

[85]  Gregory R. Madey,et al.  Verification and Validation of Agent-based Scientific Simulation Models , 2005 .

[86]  Joan M. Francioni,et al.  Debugging parallel programs using sound , 1991, PADD '91.

[87]  Adelinde M. Uhrmacher,et al.  VARIABLE STRUCTURE MODELS IN OBJECT-ORIENTED SIMULATION∗ , 1996 .

[88]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[89]  John A. Tufarolo,et al.  A case study of verification, validation, and accreditation for advanced distributed simulation , 1997, TOMC.

[90]  Robert E. Shannon,et al.  Introduction to the art and science of simulation , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[91]  Rémy Courdier,et al.  Agent-based simulation of complex systems: application to collective management of animal wastes , 2002, J. Artif. Soc. Soc. Simul..

[92]  Robert G. Sargent A historical view of hybrid simulation/analytic models , 1994, Proceedings of Winter Simulation Conference.

[93]  Niclas Feldkamp,et al.  Online Analysis of Simulation Data with Stream-based Data Mining , 2017, SIGSIM-PADS.

[94]  Joshua M. Epstein,et al.  Agent-based computational models and generative social science , 1999, Complex..

[95]  Pei-Ning Wang,et al.  Comorbidity and dementia: A nationwide survey in Taiwan , 2017, PloS one.

[96]  James R. Wilson,et al.  A brief history of simulation revisited , 2010, Proceedings of the 2010 Winter Simulation Conference.

[97]  Giancarlo Mauri,et al.  Integration of single-cell RNA-seq data into population models to characterize cancer metabolism , 2019, PLoS Comput. Biol..

[98]  Tim Storer,et al.  Bridging the Chasm , 2017, ACM Comput. Surv..

[99]  Stephen Brewster Multimodal Feedback Cues To Aid Veterinary Training Simulations , 2001 .

[100]  Ross Gore,et al.  Modelling terror management theory: computer simulations of the impact of mortality salience on religiosity , 2018 .

[101]  Lee W. Schruben,et al.  Establishing the credibility of simulations , 1980 .

[102]  John A. Sokolowski,et al.  Effective visualization in modeling & simulation , 2015, SpringSim.

[103]  Azzedine Boukerche,et al.  Towards the design of an interoperable multi-cloud distributed simulation system , 2017, SpringSim.

[104]  Osman Balci,et al.  Formulated problem verification as an explicit requirement of model credibility , 1985 .

[105]  Sara A. Bly,et al.  Communicating with sound (panel session , 1985, CHI '85.

[106]  D. A. Duce,et al.  Visualization in Scientific Computing , 1994, Focus on Computer Graphics.

[107]  Hans J. W. Spoelder,et al.  Visualization of time-dependent data with feature tracking and event detection , 2001, The Visual Computer.

[108]  Christopher J. Lynch,et al.  Incorporating sound in simulations , 2017, 2017 Winter Simulation Conference (WSC).

[109]  A. Saltelli,et al.  On the Relative Importance of Input Factors in Mathematical Models , 2002 .

[110]  Stewart Robinson,et al.  Simulation model verification and validation: increasing the users' confidence , 1997, WSC '97.

[111]  Aniruddha S. Gokhale,et al.  Cloud-hosted simulation-as-a-service for high school STEM education , 2015, Simul. Model. Pract. Theory.

[112]  Robert G. Sargent Some subjective validation methods using graphical displays of data , 1996, Winter Simulation Conference.

[113]  Ingo J. Timm,et al.  A survey on methodological aspects of computer simulation as research technique , 2015, 2015 Winter Simulation Conference (WSC).

[114]  Garth R. MacKenzie,et al.  Verification Technology Potential with Different Modeling and Simulation Development and Implementation Paradigms , 2002 .

[115]  Winfried Kurth,et al.  Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and 'R' , 2014, J. Artif. Soc. Soc. Simul..

[116]  Eddy R. Segura,et al.  Anticipated Notification of Sexual Partners following STD Diagnosis among Men Who Have Sex with Men and Transgender Women in Lima, Peru: A Mixed Methods Analysis , 2016, PloS one.

[117]  Robert Axelrod,et al.  Advancing the art of simulation in the social sciences , 1997, Complex..

[118]  William Becker,et al.  Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices , 2017, Environ. Model. Softw..

[119]  Paul Vickers,et al.  Siren songs and swan songs debugging with music , 2003, CACM.

[120]  Christopher J. Lynch,et al.  An overview of modeling and simulation using content analysis , 2015, Scientometrics.

[121]  Adelinde M. Uhrmacher,et al.  Dynamic structures in modeling and simulation: a reflective approach , 2001, TOMC.

[122]  Steffen Strassburger,et al.  Visualization and Interaction for Knowledge Discovery in Simulation Data , 2020, HICSS.

[123]  Leland Wilkinson,et al.  The History of the Cluster Heat Map , 2009 .

[124]  Christopher J. Lynch,et al.  Towards a World Wide Web of Simulation , 2017 .

[125]  Andreas Tolk,et al.  Hybrid simulation studies and Hybrid Simulation systems: Definitions, challenges, and benefits , 2015, 2015 Winter Simulation Conference (WSC).

[126]  Bruce W. Schmeiser,et al.  Some myths and common errors in simulation experiments , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[127]  Philip E. Bourne,et al.  Ten Simple Rules for Better Figures , 2014, PLoS Comput. Biol..

[128]  Jim Duggan,et al.  Using R libraries to facilitate sensitivity analysis and to calibrate system dynamics models , 2019, System Dynamics Review.

[129]  Priscilla Glasgow,et al.  Simulation Validation (SIMVAL) 1999, Making VV&A Effective and Affordable Mini-Symposium and Workshop , 1999 .

[130]  Alice Mannocci,et al.  Mediterranean diet adherence and synergy with acute myocardial infarction and its determinants: A multicenter case-control study in Italy , 2018, PloS one.

[131]  James R. Wilson,et al.  A retrospective oral history of computer simulation: Progress report , 2013, 2013 Winter Simulations Conference (WSC).

[132]  Richard W. Conway,et al.  Some Tactical Problems in Digital Simulation , 1963 .

[133]  Osman Balci,et al.  Verification, Validation, and Testing , 2007 .

[134]  Steven E. Stemler,et al.  An Overview of Content Analysis. , 2001 .

[135]  William Becker,et al.  Design for Sensitivity Analysis , 2012 .

[136]  Matthias Rauterberg About the importance of auditory alarms during the operation of a plant simulator , 1998, Interact. Comput..

[137]  Mengchen Liu,et al.  A survey on information visualization: recent advances and challenges , 2014, The Visual Computer.

[138]  Sogol Kharrazi,et al.  Study of industrially applied methods for verification, validation and uncertainty quantification of simulator models , 2015, Int. J. Model. Simul. Sci. Comput..

[139]  T. Huynh,et al.  Large Domain Motions in Ago Protein Controlled by the Guide DNA-Strand Seed Region Determine the Ago-DNA-mRNA Complex Recognition Process , 2013, PloS one.

[140]  Paul Vickers,et al.  Musical program auralization: Empirical studies , 2005, TAP.

[141]  Daniel M. McNeish,et al.  Clustered data with small sample sizes: Comparing the performance of model-based and design-based approaches , 2017, Commun. Stat. Simul. Comput..

[142]  Thomas H. Naylor,et al.  Verification of Computer Simulation Models , 1967 .

[143]  Osman Balci,et al.  A methodology for cost-risk analysis in the statistical validation of simulation models , 1981, CACM.

[144]  Robert G Sargent,et al.  An Assessment Procedure and a Set of Criteria for Use in the Evaluation of 'Computerized Models and Computer-Based Modelling Tools'. , 1981 .

[145]  S. Diallo,et al.  Temporal and spatiotemporal investigation of tourist attraction visit sentiment on Twitter , 2018, PloS one.

[146]  D. Morris Max,et al.  Design of Computer Experiments: Introduction and Background , 2015 .

[147]  Sarah K. Brem,et al.  Using Critical Thinking To Conduct Effective Searches of Online Resources. ERIC Digest. , 2000 .

[148]  John Cullyer,et al.  Safety critical systems , 1993, Microprocess. Microsystems.

[149]  Francky Catthoor,et al.  Classification of Resilience Techniques Against Functional Errors at Higher Abstraction Layers of Digital Systems , 2017, ACM Comput. Surv..

[150]  Jonathan P. Bowen,et al.  Safety-critical systems, formal methods and standards , 1993, Softw. Eng. J..

[151]  Osman Balci,et al.  Validation fo Multivariate Response Trace-driven Simulation Models , 1983, Performance.

[152]  Gary B. Lamont,et al.  Evolutionary sensor allocation for the Space Surveillance Network , 2018 .

[153]  Christopher J. Lynch,et al.  Content analysis to classify and compare Live, Virtual, Constructive simulations and System of Systems , 2016 .

[154]  Paul Windrum,et al.  Empirical Validation of Agent-Based Models: Alternatives and Prospects , 2007, J. Artif. Soc. Soc. Simul..

[155]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[156]  H. H. Kassarjian Content Analysis in Consumer Research , 1977 .

[157]  George B. Kleindorfer,et al.  Validation in Simulation: Various Positions in the Philosophy of Science , 1998 .

[158]  Christopher J. Lynch,et al.  A Generative Model of the Mutual Escalation of Anxiety Between Religious Groups , 2018, J. Artif. Soc. Soc. Simul..

[159]  Gabriel Zachmann,et al.  Virtual reality as a tool for verification of assembly and maintenance processes , 1999, Comput. Graph..

[160]  James R. Wilson,et al.  A brief history of simulation , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[161]  M. Stein Large sample properties of simulations using latin hypercube sampling , 1987 .

[162]  Jac A. M. Vennix,et al.  Group model-building: tackling messy problems , 1999 .

[163]  Osman Balci Validation, verification, and testing techniques throughout the life cycle of a simulation study , 1994, WSC '94.

[164]  Andreas Tolk,et al.  Reference modelling in support of M&S—foundations and applications , 2013, J. Simulation.

[165]  Stewart Robinson,et al.  Conceptual Modeling for Discrete-Event Simulation , 2010 .

[166]  Richard A. Malinauskas,et al.  Use of the FDA nozzle model to illustrate validation techniques in computational fluid dynamics (CFD) simulations , 2017, PloS one.

[167]  Richard E. Nance,et al.  Hybrid Simulation: Historical lessons, present challenges and futures , 2016, 2016 Winter Simulation Conference (WSC).

[168]  John A. Sokolowski,et al.  Modeling and Simulation Fundamentals: Theoretical Underpinnings and Practical Domains , 2010 .

[169]  Richard E. Nance,et al.  Perspectives on the Evolution of Simulation , 2002, Oper. Res..

[170]  William W. Gaver,et al.  Effective sounds in complex systems: the ARKOLA simulation , 1991, CHI.

[171]  Jack P. C. Kleijnen,et al.  Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models , 1997 .

[172]  James L. Alty,et al.  Can We Use Music in Computer-Human Communication? , 1996, BCS HCI.

[173]  Ross Gore,et al.  Statistical Debugging for Simulations , 2015, ACM Trans. Model. Comput. Simul..

[174]  Christopher J. Lynch,et al.  HOW TO CREATE EMPATHY AND UNDERSTANDING: NARRATIVE ANALYTICS IN AGENT-BASED MODELING , 2018, 2018 Winter Simulation Conference (WSC).

[175]  Per Runeson,et al.  Verification and validation in industry - a qualitative survey on the state of practice , 2002, Proceedings International Symposium on Empirical Software Engineering.

[176]  Osman Balci,et al.  History of verification and validation of simulation models , 2017, 2017 Winter Simulation Conference (WSC).

[177]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[178]  Rui L. Aguiar,et al.  Manycore simulation for peta-scale system design: Motivation, tools, challenges and prospects , 2017, Simul. Model. Pract. Theory.