A pattern-based validation method for the credibility evaluation of simulation models

Measuring the credibility of a simulation model has always been challenging due to its growing uncertainty and complexity. During the past decades, plenty of metrics and evaluation procedures have been developed for evaluating different sorts of simulation models. Most of the existing research focuses on the direct comparison of numerical results with a group of reference data. However, it is sometimes unsuitable for evolving dynamic models such as the multi-agent models. With the same condition, both the practical system and the simulation model perform highly dynamic actions. The credibility of the model with insufficient information, non-stationary states and changing environment is unable to acquire through a direct pair comparison. This paper presents a pattern-based validation method to complementarily extract hidden patterns that exist in both a simulation model and its reference data, and assess the model performance in a different aspect. Firstly, multi-dimensional perceptually important points strategy is modified to find the patterns exist in time-serial data. Afterward, a pattern organizing topology is applied to automatically depict required pattern from reference data and assess the performance of the corresponding simulation model. The extensive case study on three simulation models shows the effectiveness of the proposed method.

[1]  Shen Furao,et al.  An enhanced self-organizing incremental neural network for online unsupervised learning , 2007, Neural Networks.

[2]  Daniel Lemire,et al.  Faster retrieval with a two-pass dynamic-time-warping lower bound , 2008, Pattern Recognit..

[3]  Bo Wen,et al.  Verification, Validation and Uncertainty Quantification (VV&UQ) Framework Applicable to Power Electronics Systems , 2014 .

[4]  David Goldsman,et al.  A tutorial on the operational validation of simulation models , 2016, 2016 Winter Simulation Conference (WSC).

[5]  Hui Liu,et al.  High dynamic adaptive mobility network model and performance analysis , 2008, Science in China Series F: Information Sciences.

[6]  Wei Chen,et al.  Toward a Better Understanding of Model Validation Metrics , 2011 .

[7]  M. V. Panduranga Rao,et al.  Probabilistic Model Checking of Incomplete Models , 2016, ISoLA.

[8]  John A. Sokolowski,et al.  Modeling and Simulation Fundamentals: Theoretical Underpinnings and Practical Domains , 2010 .

[9]  Michal Jakob,et al.  VALFRAM: Validation Framework for Activity-Based Models , 2016, J. Artif. Soc. Soc. Simul..

[10]  Segismundo S. Izquierdo,et al.  The option to leave: conditional dissociation in the evolution of cooperation. , 2010, Journal of theoretical biology.

[11]  Erdem Acar,et al.  Effect of error metrics on optimum weight factor selection for ensemble of metamodels , 2015, Expert Syst. Appl..

[12]  Douglas C. Schmidt,et al.  Guest Editor's Introduction: Model-Driven Engineering , 2006, Computer.

[13]  Douglas G. Altman,et al.  Adequate sample size for developing prediction models is not simply related to events per variable , 2016, Journal of clinical epidemiology.

[14]  HeMing Zhang.  A solution of multidisciplinary collaborative simulation for complex engineering systems in a distributed heterogeneous environment , 2009, Science in China Series F: Information Sciences.

[15]  Nurul I. Sarkar,et al.  Revisiting the issue of the credibility of simulation studies in telecommunication networks: highlighting the results of a comprehensive survey of IEEE publications , 2014, IEEE Communications Magazine.

[16]  Ming Yang,et al.  Study on simulation credibility metrics , 2005, Proceedings of the Winter Simulation Conference, 2005..

[17]  Jaemyung Ahn,et al.  Credibility Assessment of Models and Simulations Based on NASA’s Models and Simulation Standard Using the Delphi Method , 2014, Syst. Eng..

[18]  Sean T. Smith,et al.  Application of a Verification, Validation and Uncertainty Quantification Framework to a Turbulent Buoyant Helium Plume , 2015 .

[19]  Philippe Schnoebelen,et al.  Systems and Software Verification, Model-Checking Techniques and Tools , 2001 .

[20]  Ross Gore,et al.  INSIGHT: understanding unexpected behaviours in agent-based simulations , 2010, J. Simulation.

[21]  William Ho,et al.  The state-of-the-art integrations and applications of the analytic hierarchy process , 2017, Eur. J. Oper. Res..

[22]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[23]  Sankaran Mahadevan,et al.  Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems , 2015, Reliab. Eng. Syst. Saf..

[24]  Averill M. Law,et al.  How to build valid and credible simulation models , 2008, 2008 Winter Simulation Conference.

[25]  Osman Balci,et al.  How to successfully conduct large-scale modeling and simulation projects , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[26]  Averill M. Law,et al.  Statistical analysis of simulation output data: the practical state of the art , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[27]  Magnus Carlsson,et al.  Enabling Uncertainty Quantification of Large Aircraft System Simulation Models , 2013 .

[28]  AhnJaemyung,et al.  Credibility Assessment of Models and Simulations Based on NASA's Models and Simulation Standard Using the Delphi Method , 2014 .

[29]  Michal Jakob,et al.  Data Driven Validation Framework for Multi-agent Activity-Based Models , 2015, MABS.

[30]  Bernard P. Zeigler,et al.  Towards a framework for more robust validation and verification of simulation models for systems of systems , 2016 .

[31]  Mario Kolberg,et al.  Verification & Validation of Agent Based Simulations using the VOMAS (Virtual Overlay Multi-agent System) Approach , 2009, MALLOW.

[32]  Joshua L. Hegenderfer,et al.  Resource Allocation Framework: Validation of Numerical Models of Complex Engineering Systems against Physical Experiments , 2012 .

[33]  Ali Mosleh,et al.  Bayesian Treatment of Model Uncertainty for Partially Applicable Models , 2014, Risk analysis : an official publication of the Society for Risk Analysis.

[34]  Tak-Chung Fu,et al.  Pattern discovery from stock time series using self-organizing maps , 2016 .

[35]  Christoph Scholl,et al.  Symbolic Model Checking for Incomplete Designs with Flexible Modeling of Unknowns , 2013, IEEE Transactions on Computers.

[36]  Osman Balci.  Credibility assessment of simulation results , 1986, WSC '86.

[37]  Jing Liu,et al.  CoSMSOL: Complex system modeling, simulation and optimization language , 2017, Int. J. Model. Simul. Sci. Comput..

[38]  Tzung-Pei Hong,et al.  Time series pattern discovery by a PIP-based evolutionary approach , 2013, Soft Comput..

[39]  Felician Campean,et al.  Application of permutation genetic algorithm for sequential model building–model validation design of experiments , 2016, Soft Comput..

[40]  J. Gabriel.  The Defense , 2013 .

[41]  Xiong Xiao,et al.  A Load-Balancing Self-Organizing Incremental Neural Network , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[42]  Vic Barnett,et al.  Sample Survey Principles and Methods , 1991 .

[43]  Gary S Collins,et al.  Sample size considerations for the external validation of a multivariable prognostic model: a resampling study , 2015, Statistics in medicine.

[44]  Donald J. Berndt,et al.  Using Dynamic Time Warping to Find Patterns in Time Series , 1994, KDD Workshop.

[45]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[46]  Dean Karnopp,et al.  System Dynamics: Modeling, Simulation, and Control of Mechatronic Systems , 1999 .

[47]  Salim Hariri,et al.  Resilient and Trustworthy Dynamic Data-driven Application Systems (DDDAS) Services for Crisis Management Environments , 2015, ICCS.

[48]  Robert G. Sargent,et al.  An interval statistical procedure for use in validation of simulation models , 2015, J. Simulation.

[49]  Shen Furao,et al.  An incremental network for on-line unsupervised classification and topology learning , 2006, Neural Networks.

[50]  Joana Hora Martins,et al.  A review of performance criteria to validate simulation models , 2015, Expert Syst. J. Knowl. Eng..

[51]  Douglas C. Schmidt,et al.  Model-Driven Engineering , 2006 .

[52]  Philippe Caillou.  Automated Multi-agent Simulation Generation and Validation , 2010, PRIMA.

[53]  Zheng Xie,et al.  A novel intelligent verification platform based on a structured analysis model , 2013, Science China Information Sciences.

[54]  Muaz A. Niazi,et al.  Towards a novel unified framework for developing formal, network and validated agent-based simulation models of complex adaptive systems , 2011, ArXiv.

[55]  Michael R. Lowry,et al.  A Credibility Assessment Scoring ( CAS ) Process for Mission Risk Management , 2011 .

[56]  Marta Z. Kwiatkowska,et al.  Automated Verification Techniques for Probabilistic Systems , 2011, SFM.

[57]  Christopher J. Roy,et al.  A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing , 2011 .

[58]  Elizabeth Sklar,et al.  NetLogo, a Multi-agent Simulation Environment , 2007, Artificial Life.

[59]  Vicente J. Romero,et al.  Simulation Credibility: Advances in Verification, Validation, and Uncertainty Quantification , 2016 .

[60]  Shane McIntosh,et al.  An Empirical Comparison of Model Validation Techniques for Defect Prediction Models , 2017, IEEE Transactions on Software Engineering.

[61]  Saul I. Gass,et al.  Model accreditation: A rationale and process for determining a numerical rating , 1993 .

[62]  Sankaran Mahadevan,et al.  Dynamics Model Validation Using Time-Domain Metrics , 2017 .

[63]  Bernard P. Zeigler,et al.  Service-Oriented Model Engineering and Simulation for System of Systems Engineering , 2015 .

[64]  Charles M. Macal,et al.  Lessons from a comprehensive validation of an agent based-model: The experience of the Pampas Model of Argentinean agricultural systems , 2014 .

[65]  Osman Balci,et al.  Verification, Validation And Accreditation Of Simulation Models , 1997, Winter Simulation Conference Proceedings,.

[66]  Tieju Ma,et al.  Assessing historical reliability of the agent-based model of the global energy system , 2016 .

[67]  Lin Zhang,et al.  Design and Developmental Research on the VV&A of Complex Simulation System , 2016 .

[68]  David Goldsman,et al.  Use of the interval statistical procedure for simulation model validation , 2015, 2015 Winter Simulation Conference (WSC).

[69]  Min Ouyang,et al.  Review on modeling and simulation of interdependent critical infrastructure systems , 2014, Reliab. Eng. Syst. Saf..

[70]  Frank L. Lewis,et al.  Aircraft control and simulation: Dynamics, controls design, and autonomous systems: Third edition , 2015 .

[71]  Luca Spalazzi,et al.  Model Checking Semantically Annotated Services , 2012, IEEE Transactions on Software Engineering.

[72]  Damian M. Lyons,et al.  Performance Verification for Behavior-Based Robot Missions , 2015, IEEE Trans. Robotics.