AutoSimOA: a framework for automated analysis of simulation output

There are two key issues in assuring the accuracy of estimates of performance obtained from a simulation model. The first is the removal of any initialisation bias; the second is ensuring that enough output data are produced to obtain an accurate estimate of performance. Our aim is to produce an automated procedure for inclusion into commercial simulation software to address both of these issues. This paper describes the results of a 3-year project to produce such an analyser. Our Automated Simulation Output Analyser identifies the warm-up period, estimates the number of replications, and/or analyses output from a single run, with the aim of providing the user with accurate and precise measures of their chosen output statistics.

[1]  M. Kendall,et al.  Kendall's advanced theory of statistics , 1995 .

[2]  W. David Kelton,et al.  A Procedure for Generating Batch-Means Confidence Intervals for Simulation: Checking Independence and Normality , 2007, Simul..

[3]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[4]  Ruth Davies,et al.  Automated selection of the number of replications for a discrete-event simulation , 2010, J. Oper. Res. Soc..

[5]  S. Pollock,et al.  Weighted Batch Means for Confidence Intervals in Steady-State Simulations , 1993 .

[6]  Ruth Davies,et al.  Automating warm-up length estimation , 2008, 2008 Winter Simulation Conference.

[7]  Thomas J. Schriber,et al.  Interactive analysis of simulation output by the method of batch means , 1979, WSC '79.

[8]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems, II: Markov Chains , 1974, JACM.

[9]  Averill M. Law,et al.  Confidence Intervals for Steady-State Simulations II: A Survey of Sequential Procedures , 1982 .

[10]  Averill M. Law,et al.  Simulation-based optimization , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[11]  Emily K. Lada,et al.  ASAP3: a batch means procedure for steady-state simulation analysis , 2005, TOMC.

[12]  W. David Kelton,et al.  Determining simulation run length with the runs test , 2003, Simul. Model. Pract. Theory.

[13]  James R. Wilson,et al.  SBatch: A spaced batch means procedure for steady-state simulation analysis , 2008 .

[14]  J. Neumann.  Distribution of the Ratio of the Mean Square Successive Difference to the Variance , 1941 .

[15]  A. Stuart,et al.  Kendall's Advanced Theory of Statistics, Volume 1: Distribution Theory , 1988 .

[16]  George S. Fishman,et al.  LABATCH.2: software for statistical analysis of simulation sample path data , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[17]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems: III. Regenerative Processes and Discrete-Event Simulations , 1975, Oper. Res..

[18]  A. J. Lemoine,et al.  The regenerative method , 1977 .

[19]  Averill M. Law,et al.  Confidence Intervals for Steady-State Simulations: I. A Survey of Fixed Sample Size Procedures , 1984, Oper. Res..

[20]  K. Preston White,et al.  A comparison of five steady-state truncation heuristics for simulation , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[21]  Michael C. Fu,et al.  Optimization for Simulation: Theory vs. Practice , 2002 .

[22]  Stewart Robinson,et al.  Soft with a hard centre: discrete-event simulation in facilitation , 2001, J. Oper. Res. Soc..

[23]  George S. Fishman,et al.  Estimating Sample Size in Computing Simulation Experiments , 1971 .

[24]  G. S. Fishman.  Grouping Observations in Digital Simulation , 1978 .

[25]  Stewart Robinson,et al.  Simulation: The Practice of Model Development and Use , 2004 .

[26]  C. D. Kemp,et al.  Kendall's Advanced Theory of Statistics, Volume 1, Distribution Theory. , 1988 .

[27]  Bruce W. Schmeiser,et al.  Batch Size Effects in the Analysis of Simulation Output , 1982, Oper. Res..

[28]  Emily K. Lada,et al.  A wavelet-based spectral procedure for steady-state simulation analysis , 2006, Eur. J. Oper. Res..

[29]  M. Sherman.  On batch means in the simulation and statistics communities , 1995, Winter Simulation Conference Proceedings, 1995..

[30]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[31]  James G. Mitchell,et al.  A comparison of two network-based file servers , 1982, CACM.

[32]  John W. Fowler,et al.  Proceedings of the 2008 Winter Simulation Conference, Global Gateway to Discovery, WSC 2008, InterContinental Hotel, Miami, Florida, USA, December 7-10, 2008 , 2008, Online World Conference on Soft Computing in Industrial Applications.

[33]  George S. Fishman,et al.  Computational experience with the batch means method , 1997, WSC '97.

[34]  van der,et al.  Proceedings of the 2012 winter simulation conference , 2001, WSC 2008.

[35]  D. Goldsman,et al.  ASAP2: an improved batch means procedure for simulation output analysis , 2002, Proceedings of the Winter Simulation Conference.

[36]  Brian W. Hollocks,et al.  Discrete-event simulation: an inquiry into user practice , 2001, Simul. Pract. Theory.

[37]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems, I: General Multiserver Queues , 1974, JACM.

[38]  George S. Fishman,et al.  Achieving specific accuracy in simulation output analysis , 1977, CACM.

[39]  C. H. Sauer,et al.  Sequential stopping rules for the regenerative method of simulation , 1977 .

[40]  Michael C. Fu,et al.  Feature Article: Optimization for simulation: Theory vs. Practice , 2002, INFORMS J. Comput..

[41]  Christos Alexopoulos,et al.  A Comprehensive Review of Methods for Simulation Output Analysis , 2006, Proceedings of the 2006 Winter Simulation Conference.

[42]  George S. Fishman,et al.  An Implementation of the Batch Means Method , 1997, INFORMS J. Comput..

[43]  Barry L. Nelson,et al.  Recent advances in ranking and selection , 2007, 2007 Winter Simulation Conference.

[44]  Averill M. Law,et al.  A Sequential Procedure for Determining the Length of a Steady-State Simulation , 1979, Oper. Res..

[45]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[46]  Barry L. Nelson,et al.  Statistical Analysis of Simulation Results , 2007 .

[47]  Philip Heidelberger,et al.  A spectral method for confidence interval generation and run length control in simulations , 1981, CACM.

[48]  Richard W. Conway,et al.  Some Tactical Problems in Digital Simulation , 1963 .

[49]  James R. Wilson,et al.  Improved batching for confidence interval construction in steady-state simulation , 1999, WSC '99.

[50]  K. Preston White,et al.  An Effective Truncation Heuristic for Bias Reduction in Simulation Output , 1997, Simul..

[51]  W. G. Cochran,et al.  The distribution of quadratic forms in a normal system, with applications to the analysis of covariance , 1934, Mathematical Proceedings of the Cambridge Philosophical Society.