Automating warm-up length estimation

AbstractThere are two key issues in assuring the accuracy of estimates of performance obtained from a simulation model. The first is the removal of any initialisation bias, the second is ensuring that enough output data is produced to obtain an accurate estimate of performance. This paper is concerned with the first issue, and more specifically warm-up estimation. Our aim is to produce an automated procedure, for inclusion into commercial simulation software, for estimating the length of warm-up and hence removing initialisation bias from simulation output data. This paper describes the extensive literature search that was carried out in order to find and assess the various existing warm-up methods, the process of short-listing and testing of candidate methods. In particular it details the extensive testing of the warm-up MSER-5 method.

[1]  Ihsan Sabuncuoglu,et al.  Analysis of the behavior of the transient period in non-terminating simulations , 2006, Eur. J. Oper. Res..

[2]  George S. Fishman,et al.  Discrete-Event Simulation : Modeling, Programming, and Analysis , 2001 .

[3]  Emily K. Lada,et al.  Performance evaluation of a wavelet-based spectral method for steady-state simulation analysis , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[4]  Averill M. Law,et al.  Simulation modelling and analysis , 1991 .

[5]  Robert G. Sargent,et al.  Statistical analysis of simulation output data , 1976, SIML.

[6]  Lee W. Schruben,et al.  Optimal Tests for Initialization Bias in Simulation Output , 1983, Oper. Res..

[7]  Kenneth W. Bauer,et al.  Initial data truncation for univariate output of discrete-event simulations using the Kalman filter , 1996 .

[8]  O. Kitao,et al.  I: METHODOLOGY , 2003, Deception: Counterdeception and Counterintelligence.

[9]  E.K. Lada,et al.  A wavelet-based spectral method for steady-state simulation analysis , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[10]  K. Preston White,et al.  An Effective Truncation Heuristic for Bias Reduction in Simulation Output , 1997, Simul..

[11]  Catherine M. Harmonosky,et al.  A comparison of selective initialization bias elimination methods , 2002, Proceedings of the Winter Simulation Conference.

[12]  James R. Wilson,et al.  A survey of research on the simulation startup problem , 1978 .

[13]  Richard W. Conway,et al.  Some Tactical Problems in Digital Simulation , 1963 .

[14]  K. Preston White,et al.  A comparison of five steady-state truncation heuristics for simulation , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[15]  Wolfgang Kreutzer,et al.  System simulation , 1986 .

[16]  Averill M. Law,et al.  Feature Article - Statistical Analysis of Simulation Output Data , 1983, Oper. Res..

[17]  H. Redkey,et al.  A new approach. , 1967, Rehabilitation record.

[18]  F. Bause,et al.  Truncation point estimation using multiple replications in parallel , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[19]  George S. Fishman,et al.  Discrete-event simulation , 2001 .

[20]  Delbert L. Kimbler,et al.  A survey of current methods for the elimination of initialization bias in digital simulation , 1987, ANSS '87.

[21]  Pierre L'Ecuyer,et al.  Good Parameters and Implementations for Combined Multiple Recursive Random Number Generators , 1999, Oper. Res..

[22]  Charles R. Cash,et al.  Evaluation of tests for initial-condition bias , 1992, WSC '92.

[23]  Young-Hae Lee,et al.  On-line determination of steady state in simulation outputs , 1997 .

[24]  James R. Wilson,et al.  Evaluation of startup policies in simulation experiments , 1978 .

[25]  C. J. Ancker,et al.  Evaluation of commonly used rules for detecting “steady state” in computer simulation , 1978 .

[26]  Enver Yücesan,et al.  Randomization tests for initialization bias in simulation output , 1993 .

[27]  J. J. Swain,et al.  Tests for transient means in simulated time series , 1994 .

[28]  Jorge Haddock,et al.  Estimating the steady-state mean from short transient simulations , 2005, Eur. J. Oper. Res..

[29]  George Vassilacopoulos.  Testing for initialization bias in simulation output , 1989, Simul..

[30]  Ricki G. Ingalls,et al.  Evaluation of methods used to detect warm-up period in steady state simulation , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[31]  Lee W. Schruben,et al.  Detecting Initialization Bias in Simulation Output , 1982, Oper. Res..

[32]  Averill M. Law,et al.  A new approach for dealing with the startup problem in discrete event simulation , 1983 .

[33]  Donald L. Iglehart,et al.  A new initial bias deletion rule , 1987, WSC '87.

[34]  K. Preston White,et al.  Determining a warm-up period for a telephone network routing simulation , 1999, WSC '99.

[35]  Emily Roth.  The relaxation time heuristic for the initial transient problem in M/M/k queueing systems , 1994 .

[36]  Emily Roth,et al.  A relaxation time heuristic for exponential-erlang queueing systems , 1993, Comput. Oper. Res..

[37]  Paul Bratley,et al.  A guide to simulation , 1983 .

[38]  Manuel D. Rossetti,et al.  Control of initialization bias in queueing simulations using queueing approximations , 1995, WSC '95.

[39]  Krzysztof Pawlikowski,et al.  Steady-state simulation of queueing processes: survey of problems and solutions , 1990, CSUR.

[40]  Xiping Ma,et al.  A Comparison Study of Two Tests for Detecting Initialization Bias in Simulation Output , 1993, Simul..

[41]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[42]  Paul T. Jackway,et al.  A Methodology for Initialisation Bias Reduction in Computer Simulation Output , 1992 .

[43]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[44]  Young Hae Lee,et al.  Detecting truncation point in steady-state simulation using chaos theory , 1994, Proceedings of Winter Simulation Conference.

[45]  Zhe Li,et al.  Exploring exponentially weighted moving average control charts to determine the warm-up period , 2005, Proceedings of the Winter Simulation Conference, 2005..

[46]  George S. Fishman,et al.  Estimating Sample Size in Computing Simulation Experiments , 1971 .

[47]  Stewart Robinson,et al.  Simulation: The Practice of Model Development and Use , 2004 .

[48]  David Goldsman,et al.  Student t-tests and compound tests to detect transients in simulated time series , 1999, Eur. J. Oper. Res..

[49]  Stewart Robinson,et al.  A statistical process control approach to selecting a warm-up period for a discrete-event simulation , 2007, Eur. J. Oper. Res..

[50]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[51]  George S. Fishman,et al.  Solution of Large Networks by Matrix Methods , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[52]  Emily K. Lada,et al.  A wavelet-based spectral procedure for steady-state simulation analysis , 2006, Eur. J. Oper. Res..