A review of performance criteria to validate simulation models

This study reviews performance criteria adequate to validate simulation models through the comparison of two quantitative data sets, concerning historical and simulated data. The criteria reviewed were organized according to its characteristics into the groups: error-based measures, information theory measures, information criteria, parametric tests, non-parametric tests, distance-based measures and combined measures. Each criterion is reviewed through its mathematic definition, its applications in literature and the identification of its advantages and drawbacks. The features assessed by each criterion are identified and discussed. This study provides a concise outline over the criteria reviewed, which can be used as a guide to help developers of simulation models into the decision on the most appropriate criteria to validate their models.

[1]  D. Weakliem.  A Critique of the Bayesian Information Criterion for Model Selection , 1999 .

[2]  Claude E. Shannon,et al.  A mathematical theory of communication , 1948, MOCO.

[3]  F. Bliemel.  Theil's Forecast Accuracy Coefficient: A Clarification , 1973 .

[4]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[5]  J. H. Zar,et al.  Significance Testing of the Spearman Rank Correlation Coefficient , 1972 .

[6]  Richard Taylor.  Interpretation of the Correlation Coefficient: A Basic Review , 1990 .

[7]  T. Kuhn,et al.  The Structure of Scientific Revolutions , 1963 .

[8]  Rob J. Hyndman,et al.  Another Look at Forecast Accuracy Metrics for Intermittent Demand , 2006 .

[9]  C. Willmott.  ON THE VALIDATION OF MODELS , 1981 .

[10]  Yang Wang,et al.  A Comparison of Mamdani and Sugeno Fuzzy Inference Systems for Traffic Flow Prediction , 2014, J. Comput..

[11]  Wentian Li.  Mutual information functions versus correlation functions , 1990 .

[12]  D. Legates,et al.  Evaluating the use of “goodness‐of‐fit” Measures in hydrologic and hydroclimatic model validation , 1999 .

[13]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[14]  Vijay S Pande,et al.  Validation of Markov state models using Shannon's entropy. , 2006, The Journal of chemical physics.

[15]  Baitao Li Chang,et al.  DPF - a perceptual distance function for image retrieval , 2002, Proceedings. International Conference on Image Processing.

[16]  Jack P. C. Kleijnen,et al.  A methodology for fitting and validating metamodels in simulation , 2000, Eur. J. Oper. Res..

[17]  Jens Christian Refsgaard,et al.  Modelling guidelinesterminology and guiding principles , 2004 .

[18]  Edward Y. Chang,et al.  Discovery of a perceptual distance function for measuring image similarity , 2003, Multimedia Systems.

[19]  Jose C. Principe,et al.  Information Theoretic Learning - Renyi's Entropy and Kernel Perspectives , 2010, Information Theoretic Learning.

[20]  Fred L. Collopy,et al.  Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons , 1992 .

[21]  G. Nigel Gilbert,et al.  Simulation for the social scientist , 1999 .

[22]  Averill M. Law.  How to Build Valid and Credible Simulation Models , 2019, 2019 Winter Simulation Conference (WSC).

[23]  John D. Bredehoeft,et al.  Ground-water models cannot be validated , 1992 .

[24]  J. Kuha.  AIC and BIC , 2004 .

[25]  Behzad Moshiri,et al.  Application of temporal difference learning rules in short‐term traffic flow prediction , 2015, Expert Syst. J. Knowl. Eng..

[26]  Filiz Günes,et al.  Analysis and design of X-band Reflectarray antenna using 3-D EM-based Artificial Neural Network model , 2012, 2012 IEEE International Conference on Ultra-Wideband.

[27]  Billy M. Williams,et al.  Systematic Approach for Validating Traffic Simulation Models , 2004 .

[28]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[29]  Panos Y. Papalambros,et al.  Comparing time histories for validation of simulation models : Error measures and metrics , 2010 .

[30]  José Manuel Gutiérrez,et al.  Evolving modular networks with genetic algorithms: application to nonlinear time series , 2004, Expert Syst. J. Knowl. Eng..

[31]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: A Novel Regression Test , 1998 .

[32]  Olena Vynokurova,et al.  An adaptive learning algorithm for a wavelet neural network , 2005, Expert Syst. J. Knowl. Eng..

[33]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[34]  Frank Klawonn,et al.  Fuzzy Clustering of Short Time-Series and Unevenly Distributed Sampling Points , 2003, IDA.

[35]  D. Posada,et al.  Model selection and model averaging in phylogenetics: advantages of akaike information criterion and bayesian approaches over likelihood ratio tests. , 2004, Systematic biology.

[36]  Ricardo Colomo Palacios,et al.  Towards a framework for multiple artificial neural network topologies validation by means of statistics , 2014, Expert Syst. J. Knowl. Eng..

[37]  Bruce Mizrach,et al.  The distribution of the Theil U-statistic in bivariate normal populations , 1992 .

[38]  Sedigheh Anvari,et al.  Effect of Southern Oscillation Index and spatially distributed climate data on improving the accuracy of Artificial Neural Network, Adaptive Neuro‐Fuzzy Inference System and K‐Nearest Neighbour streamflow forecasting models , 2013, Expert Syst. J. Knowl. Eng..

[39]  Pedro Sousa,et al.  Multi‐scale Internet traffic forecasting using neural networks and time series methods , 2010, Expert Syst. J. Knowl. Eng..

[40]  Jack P. C. Kleijnen,et al.  Validation of simulation model for robotic milking barn design , 2001, Eur. J. Oper. Res..

[41]  Xiang Jiang,et al.  Bayesian cross-entropy methodology for optimal design of validation experiments , 2006 .

[42]  C. Willmott,et al.  Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance , 2005 .

[43]  J. Kleijnen.  Statistical tools for simulation practitioners , 1986 .

[44]  Hong Wan,et al.  Efficient experimental design tools for exploring large simulation models , 2009, Comput. Math. Organ. Theory.

[45]  J. Richardson,et al.  le principle d’humanité (the humanity principle) , 2002 .

[46]  Ryszard S. Michalski,et al.  Machine learning: an artificial intelligence approach volume III , 1990 .

[47]  Matthias Meyer,et al.  Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments , 2011, Computational and Mathematical Organization Theory.

[48]  Leonard E. Schwer,et al.  Validation metrics for response histories: perspectives and case studies , 2007, Engineering with Computers.

[49]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[50]  V. Klemeš,et al.  Operational Testing of Hydrological Simulation Models , 2022 .

[51]  T. Warren Liao,et al.  Clustering of time series data - a survey , 2005, Pattern Recognit..

[52]  Leon Sterling,et al.  The Art of Agent-Oriented Modeling , 2009 .

[53]  Robert G. Sargent,et al.  Validation and verification of simulation models , 1999, Proceedings of the 2004 Winter Simulation Conference, 2004..

[54]  Jack P. C. Kleijnen,et al.  Generalization of simulation results practicality of statistical methods , 1979 .

[55]  James Bailey,et al.  Information Theoretic Measures for Clusterings Comparison: Variants, Properties, Normalization and Correction for Chance , 2010, J. Mach. Learn. Res..

[56]  Panos Y. Papalambros,et al.  A Comprehensive Metric for Comparing Time Histories in Validation of Simulation Models With Emphasis on Vehicle Safety Applications , 2008, DAC 2008.