Output analysis for simulation

Methods are reviewed for analyzing data produced by simulations. The author discusses the objectives of output analysis and examines the need to apply statistical methods. Methods for estimating parameters from terminating and steady-state simulations are presented. For steady-state simulations, methods for computing estimates of the mean as well as proportions and percentiles are presented. Multivariate estimation techniques are discussed, and some practical advice is provided concerning output analysis in practice.<<ETX>>

[1]  James R. Wilson,et al.  A survey of research on the simulation startup problem , 1978 .

[2]  Averill M. Law,et al.  Confidence Intervals for Steady-State Simulations: I. A Survey of Fixed Sample Size Procedures , 1984, Oper. Res..

[3]  F. David,et al.  Statistical Techniques in Simulation: Part I , 1975 .

[4]  Averill M. Law.  Statistical Analysis of Simulation Output Data , 1980 .

[5]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems, I: General Multiserver Queues , 1974, JACM.

[6]  D. Iglehart.  Simulating stable stochastic systems, V: Comparison of ratio estimators , 1975 .

[7]  A. Law.  Statistical Analysis of the Output Data from Terminating Simulations. , 1980 .

[8]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems, II: Markov Chains , 1974, JACM.

[9]  Linus Schrage,et al.  A guide to simulation , 1983 .

[10]  Reuven Y. Rubinstein,et al.  Simulation and the Monte Carlo Method , 1981 .

[11]  Lee W. Schruben,et al.  Note-New Confidence Interval Estimators Using Standardized Time Series , 1990 .

[12]  C. J. Ancker,et al.  Evaluation of commonly used rules for detecting “steady state” in computer simulation , 1978 .

[13]  Stephen S. Lavenberg,et al.  Computer Performance Modeling Handbook , 1983, Int. CMG Conference.

[14]  George S. Fishman,et al.  Solution of Large Networks by Matrix Methods , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[15]  George S Fishman.  Confidence Intervals for a Mean and a Proportion in the Bounded Case. , 1986 .

[16]  P. Heidelberger,et al.  Adaptive spectral methods for simulation output analysis , 1981 .

[17]  Paul Bratley,et al.  A guide to simulation (2nd ed.) , 1986 .

[18]  G. S. Fishman.  Grouping Observations in Digital Simulation , 1978 .

[19]  James R. Wilson.  Variance Reduction Techniques for Digital Simulation , 1984 .

[20]  Averill M. Law,et al.  A Sequential Procedure for Determining the Length of a Steady-State Simulation , 1979, Oper. Res..

[21]  A. A. Crane,et al.  An introduction to the regenerative method for simulation analysis , 1977 .

[22]  Andrew F. Seila.  Multivariate Simulation Output Analysis , 1984 .

[23]  S. Resnick.  Adventures in stochastic processes , 1992 .

[24]  Lee W. Schruben,et al.  Confidence Interval Estimation Using Standardized Time Series , 1983, Oper. Res..

[25]  H. Robbins,et al.  ON THE ASYMPTOTIC THEORY OF FIXED-WIDTH SEQUENTIAL CONFIDENCE INTERVALS FOR THE MEAN. , 1965 .

[26]  A.F. Seila.  Output analysis for simulation , 1990, 1990 Winter Simulation Conference Proceedings.

[27]  Lee W. Schruben,et al.  Detecting Initialization Bias in Simulation Output , 1982, Oper. Res..

[28]  Averill M. Law,et al.  A new approach for dealing with the startup problem in discrete event simulation , 1983 .

[29]  George S. Fishman.  Estimation in Multiserver Queuing Simulations , 1974, Oper. Res..

[30]  Jack P. C. Kleijnen,et al.  Statistical Techniques in Simulation , 1977, IEEE Transactions on Systems, Man and Cybernetics.

[31]  Bruce W. Schmeiser,et al.  Overlapping batch means: something for nothing? , 1984, WSC '84.

[32]  Averill M. Law,et al.  Feature Article - Statistical Analysis of Simulation Output Data , 1983, Oper. Res..

[33]  L. C. Young,et al.  On Randomness in Ordered Sequences , 1941 .

[34]  George S. Fishman,et al.  Achieving specific accuracy in simulation output analysis , 1977, CACM.

[35]  Sheldon M. Ross,et al.  Introduction to Probability Models, Eighth Edition , 1972 .

[36]  L. Schruben,et al.  Asymptotic Properties of Some Confidence Interval Estimators for Simulation Output , 1984 .

[37]  A. Nádas.  An Extension of a Theorem of Chow and Robbins on Sequential Confidence Intervals for the Mean , 1969 .

[38]  James R. Wilson,et al.  Evaluation of startup policies in simulation experiments , 1978 .

[39]  Philip Heidelberger,et al.  Quantile Estimation in Dependent Sequences , 1984, Oper. Res..

[40]  Lee W. Shruben.  Control of initialization bias in multivariate simulation response , 1981 .

[41]  Jack P. C. Kleijnen,et al.  Statistical Techniques in Simulation , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[42]  Averill M. Law.  Confidence intervals in discrete event simulation: A comparison of replication and batch means , 1977 .

[43]  W. David Kelton,et al.  Random Initialization Methods in Simulation , 1989 .

[44]  G. Newman,et al.  CONFIDENCE INTERVALS , 1987, The Lancet.

[45]  Thomas J. Schriber,et al.  Interactive analysis of simulation output by the method of batch means , 1979, WSC '79.

[46]  R. H. Kent,et al.  The Mean Square Successive Difference , 1941 .

[47]  Jack P. C. Kleijnen,et al.  Statistical aspects of simulation : An updated survey , 1982 .

[48]  Andrew F. Seila,et al.  Output analysis for simulation (tutorial session) , 1990, WSC' 90.

[49]  Donald L. Iglehart,et al.  Simulation Output Analysis Using Standardized Time Series , 1990, Math. Oper. Res..

[50]  Ian T. Jolliffe,et al.  Simulation Methodology for Statisticians, Operations Analysts, and Engineers, Vol. 1. , 1990 .

[51]  Michael A. Crane,et al.  Simulating Stable Stochastic Systems: III. Regenerative Processes and Discrete-Event Simulations , 1975, Oper. Res..

[52]  Paul Bratley,et al.  A guide to simulation , 1983 .

[53]  A. J. Lemoine,et al.  The regenerative method , 1977 .

[54]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[55]  Keebom Kang,et al.  An Investigation of Finite-Sample Behavior of Confidence Interval Estimators , 1992, Oper. Res..

[56]  Andrew F. Seila,et al.  Multivariate inference in stationary simulation using batch means , 1987, WSC '87.

[57]  George S. Fishman,et al.  Bias Considerations in Simulation Experiments , 1972, Oper. Res..

[58]  Lee W. Schruben,et al.  Optimal Tests for Initialization Bias in Simulation Output , 1983, Oper. Res..

[59]  Philip Heidelberger,et al.  A spectral method for confidence interval generation and run length control in simulations , 1981, CACM.

[60]  Richard W. Conway,et al.  Some Tactical Problems in Digital Simulation , 1963 .

[61]  Chiahon Chien.  Small-sample theory for steady state confidence intervals , 1988, WSC '88.

[62]  Averill M. Law,et al.  Confidence Intervals for Steady-State Simulations II: A Survey of Sequential Procedures , 1982 .

[63]  John M. Charnes,et al.  Power comparisons for the multivariate batch-means method , 1990, 1990 Winter Simulation Conference Proceedings.

[64]  J. Neumann.  Distribution of the Ratio of the Mean Square Successive Difference to the Variance , 1941 .

[65]  Steven D. Duket,et al.  Examination of simulation output using spectral methods , 1978 .

[66]  Bruce W. Schmeiser,et al.  Batch Size Effects in the Analysis of Simulation Output , 1982, Oper. Res..

[67]  George S. Fishman,et al.  Statistical Analysis for Queueing Simulations , 1973 .

[68]  Lee W. Schruben.  Control of initialization bias in multivariate simulation response , 1981, CACM.

[69]  Irwin W. Kabak.  Stopping rules for queueing simulations: Non independent tours , 1968 .

[70]  Andrew F. Seila,et al.  Estimation of percentiles in discrete event simulation , 1982 .

[71]  A. F. Seila,et al.  A Batching Approach to Quantile Estimation in Regenerative Simulations , 1982 .

[72]  L. Schruben,et al.  Properties of standardized time series weighted area variance estimators , 1990 .

[73]  G. S. Fishman.  Principles of Discrete Event Simulation , 1978 .

[74]  John M. Charnes,et al.  Multivariate simulation output analysis , 1991, 1991 Winter Simulation Conference Proceedings..

[75]  Bruce W. Schmeiser,et al.  Properties of batch means from stationary ARMA time series , 1987 .

[76]  Irwin W. Kabak,et al.  Letter to the Editor - Stopping Rules for Queuing Simulations , 1968, Oper. Res..

[77]  Donald L. Iglehart,et al.  Simulating Stable Stochastic Systems, VI: Quantile Estimation , 1976, JACM.

[78]  R. W. Andrews,et al.  Arma-Based Confidence Intervals for Simulation Output Analysis , 1984 .

[79]  Peter D. Welch,et al.  On the relationship between batch means, overlapping means and spectral estimation , 1987, WSC '87.

[80]  P. Billingsley,et al.  Convergence of Probability Measures , 1970, The Mathematical Gazette.