Stochastic simulation under input uncertainty: A Review

Abstract Stochastic simulation is an invaluable tool for operations-research practitioners for the performance evaluation of systems with random behavior and mathematically intractable performance measures. An important step in the development of a simulation model is input modeling, which is the selection of appropriate probability models that characterize the stochastic behavior of the system inputs. For example, in a queueing-system simulation, input modeling includes choosing the probability distributions for stochastic interarrival and service times. The lack of knowledge about the true input models is an important practical challenge. The impact of the lack of information about the true input model on the simulation output is referred to as ‘input uncertainty’ in the simulation literature. Ignoring input uncertainty often leads to poor estimates of the system performance, especially when there is limited amount of historical data to make inference on the input models. Therefore, it is critically important to assess the impact of input uncertainty on the estimated performance measures in a statistically valid and computationally efficient way. The goal of this paper is to present input uncertainty research in stochastic simulations by providing a classification of major research streams and focusing on the new developments in recent years. We also review application papers that investigate the value of representing input uncertainty in the simulation of real-world stochastic systems in various industries. We provide a self-contained presentation of the major research streams with a special attention on the new developments in the last couple of years.

[1]  James R. Wilson,et al.  Accounting for input model and parameter uncertainty in simulation , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[2]  Wei Xie,et al.  Multivariate Input Uncertainty in Output Analysis for Stochastic Simulation , 2016, ACM Trans. Model. Comput. Simul..

[3]  Wei Xie,et al.  An Efficient Budget Allocation Approach for Quantifying the Impact of Input Uncertainty in Stochastic Simulation , 2017, ACM Trans. Model. Comput. Simul..

[4]  Canan G. Corlu,et al.  Simulation of inventory systems with unknown input models: a data-driven approach , 2017, Int. J. Prod. Res..

[5]  Wei Xie,et al.  A SIMULATION-BASED PREDICTION FRAMEWORK FOR STOCHASTIC SYSTEM DYNAMIC RISK MANAGEMENT , 2018, 2018 Winter Simulation Conference (WSC).

[6]  Stephen E. Chick,et al.  Bayesian Ideas and Discrete Event Simulation: Why, What and How , 2006, Proceedings of the 2006 Winter Simulation Conference.

[7]  Bahar Biller,et al.  Stochastic input model selection , 2013 .

[8]  Uday V. Shanbhag,et al.  Stochastic Approximation for simulation Optimization under Input Uncertainty with Streaming Data , 2019, 2019 Winter Simulation Conference (WSC).

[9]  C. Genest,et al.  A semiparametric estimation procedure of dependence parameters in multivariate families of distributions , 1995 .

[10]  Wei Xie,et al.  A framework for input uncertainty analysis , 2010, Proceedings of the 2010 Winter Simulation Conference.

[11]  Jayendran Venkateswaran,et al.  Robust simulation based optimization with input uncertainty , 2017, 2017 Winter Simulation Conference (WSC).

[12]  Bahar Biller,et al.  Copula-Based Multivariate Input Models for Stochastic Simulation , 2009, Oper. Res..

[13]  Bahar Biller,et al.  Improved Inventory Targets in the Presence of Limited Historical Demand Data , 2011, Manuf. Serv. Oper. Manag..

[14]  Henry Lam,et al.  Sensitivity to Serial Dependency of Input Processes: A Robust Approach , 2016, Manag. Sci..

[15]  Bahar Biller,et al.  A simulation-based approach to capturing autocorrelated demand parameter uncertainty in inventory management , 2012, Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC).

[16]  Henry Lam,et al.  Robust Sensitivity Analysis for Stochastic Systems , 2013, Math. Oper. Res..

[17]  Nando de Freitas,et al.  An Introduction to MCMC for Machine Learning , 2004, Machine Learning.

[18]  Enlu Zhou,et al.  Risk Quantification in Stochastic Simulation under Input Uncertainty , 2015, ACM Trans. Model. Comput. Simul..

[19]  Hui Xiao,et al.  Simulation Budget Allocation for Selecting the Top-m Designs With Input Uncertainty , 2018, IEEE Transactions on Automatic Control.

[20]  Russell R. Barton,et al.  Statistical Uncertainty Analysis for Stochastic Simulation , 2014 .

[21]  Canan G. Corlu,et al.  Driving inventory system simulations with limited demand data: Insights from the newsvendor problem , 2018, J. Simulation.

[22]  Bo Wang,et al.  METAMODEL-ASSISTED RISK ANALYSIS FOR STOCHASTIC SIMULATION WITH INPUT UNCERTAINTY , 2018, 2018 Winter Simulation Conference (WSC).

[23]  Henry Lam,et al.  Random Perturbation and Bagging to Quantify Input Uncertainty , 2019, 2019 Winter Simulation Conference (WSC).

[24]  Barry L. Nelson,et al.  Single-experiment input uncertainty , 2015, J. Simulation.

[25]  Russell C. H. Cheng,et al.  Selecting input models , 1994, Proceedings of Winter Simulation Conference.

[26]  Henry Lam,et al.  Subsampling to Enhance Efficiency in Input Uncertainty Quantification , 2018, Oper. Res..

[27]  Henry Lam,et al.  The empirical likelihood approach to simulation input uncertainty , 2016, 2016 Winter Simulation Conference (WSC).

[28]  Wei Xie,et al.  Simulation optimization when facing input uncertainty , 2015, 2015 Winter Simulation Conference (WSC).

[29]  Enlu Zhou,et al.  Optimizing Conditional Value-at-Risk via gradient-based adaptive stochastic search , 2016, 2016 Winter Simulation Conference (WSC).

[30]  Henry Lam,et al.  Advanced tutorial: Input uncertainty and robust analysis in stochastic simulation , 2016, 2016 Winter Simulation Conference (WSC).

[31]  Wei Xie,et al.  Statistical uncertainty analysis for stochastic simulation with dependent input models , 2014, Proceedings of the Winter Simulation Conference 2014.

[32]  Soumyadip Ghosh,et al.  Robust Analysis in Stochastic Simulation: Computation and Performance Guarantees , 2015, Oper. Res..

[33]  Szu Hui Ng,et al.  Reducing parameter uncertainty for stochastic systems , 2006, TOMC.

[34]  Weiwei Chen,et al.  A worst‐case formulation for constrained ranking and selection with input uncertainty , 2019, Naval Research Logistics (NRL).

[35]  Stephen E. Chick,et al.  Bayesian analysis for simulation input and output , 1997, WSC '97.

[36]  Szu Hui Ng,et al.  Joint criterion for factor identification and parameter estimation , 2002, Proceedings of the Winter Simulation Conference.

[37]  Russell C. H. Cheng,et al.  Calculation of confidence intervals for simulation output , 2004, TOMC.

[38]  Michael Pearce,et al.  Bayesian Optimisation vs. Input Uncertainty Reduction , 2020, ArXiv.

[39]  Canan G. Corlu,et al.  Demand fulfillment probability in a multi-item inventory system with limited historical data , 2017 .

[40]  Barry L. Nelson,et al.  Advanced tutorial: Input uncertainty quantification , 2014, Proceedings of the Winter Simulation Conference 2014.

[41]  Hui Xiao,et al.  Robust ranking and selection with optimal computing budget allocation , 2017, Autom..

[42]  Barry L. Nelson,et al.  Input Model Risk , 2017 .

[43]  Alp Akcay,et al.  Simulation-based production planning for engineer-to-order systems with random yield , 2017, 2017 Winter Simulation Conference (WSC).

[44]  Di Wu,et al.  Ranking and selection under input uncertainty: A budget allocation formulation , 2017, 2017 Winter Simulation Conference (WSC).

[45]  Canan G. Corlu,et al.  A subset selection procedure under input parameter uncertainty , 2013, 2013 Winter Simulations Conference (WSC).

[46]  B. Biller,et al.  Copula-based multivariate input modeling , 2012 .

[47]  David Fernando Muñoz,et al.  Solving the newsvendor problem under parametric uncertainty using simulation , 2015, 2015 Winter Simulation Conference (WSC).

[48]  Stephen E. Chick,et al.  Input Distribution Selection for Simulation Experiments: Accounting for Input Uncertainty , 2001, Oper. Res..

[49]  R. Cheng,et al.  Sensitivity of computer simulation experiments to errors in input data , 1997 .

[50]  Xiaowei Zhang,et al.  Distributionally Robust Selection of the Best , 2019, Manag. Sci..

[51]  David Fernando Muñoz,et al.  On the incorporation of parameter uncertainty for inventory management using simulation , 2012, Int. Trans. Oper. Res..

[52]  Michael Freimer,et al.  Collecting data and estimating parameters for input distributions , 2002, Proceedings of the Winter Simulation Conference.

[53]  Wei Xie,et al.  A sequential experiment design for input uncertainty quantification in stochastic simulation , 2015, 2015 Winter Simulation Conference (WSC).

[54]  Bahar Biller,et al.  Input uncertainty in stochastic simulations in the presence of dependent discrete input variables , 2018, J. Simulation.

[55]  Zhi Zhou,et al.  A Nonparametric Bayesian Framework for Short-Term Wind Power Probabilistic Forecast , 2019, IEEE Transactions on Power Systems.

[56]  Wei Xie,et al.  Quantifying Input Uncertainty via Simulation Confidence Intervals , 2014, INFORMS J. Comput..

[57]  Lawrence Leemis,et al.  Panel on current issues in simulation input modeling , 2002, Proceedings of the Winter Simulation Conference.

[58]  Loo Hay Lee,et al.  A Simulation Budget Allocation Procedure for Enhancing the Efficiency of Optimal Subset Selection , 2016, IEEE Transactions on Automatic Control.

[59]  Loo Hay Lee,et al.  Optimal computing budget allocation for complete ranking with input uncertainty , 2020 .

[60]  Di Wu,et al.  Solving Bayesian risk optimization via nested stochastic gradient estimation , 2020, IISE Trans..

[61]  Barry L. Nelson,et al.  Shapley Effects for Global Sensitivity Analysis: Theory and Computation , 2016, SIAM/ASA J. Uncertain. Quantification.

[62]  Loo Hay Lee,et al.  Approximate Simulation Budget Allocation for Selecting the Best Design in the Presence of Stochastic Constraints , 2012, IEEE Transactions on Automatic Control.

[63]  Barry L. Nelson,et al.  Modeling and generating multivariate time-series input processes using a vector autoregressive technique , 2003, TOMC.

[64]  Di Wu,et al.  Simulation Optimization Under Input Model Uncertainty , 2017 .

[65]  Xiaowei Zhang,et al.  Robust selection of the best , 2013, 2013 Winter Simulations Conference (WSC).

[66]  Russell R. Barton,et al.  REVISITING DIRECT BOOTSTRAP RESAMPLING FOR INPUT MODEL UNCERTAINTY , 2018, 2018 Winter Simulation Conference (WSC).

[67]  Jürgen Branke,et al.  Proceedings of the 2017 Winter Simulation Conference , 2017 .

[68]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[69]  Canan G. Corlu,et al.  Stochastic Simulation Model Development for Biopharmaceutical Production Process Risk Analysis and Stability Control , 2019, 2019 Winter Simulation Conference (WSC).

[70]  Lee W. Schruben,et al.  Resampling methods for input modeling , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[71]  Hui Xiao,et al.  Optimal computing budget allocation with input uncertainty , 2016, 2016 Winter Simulation Conference (WSC).

[72]  Canan G. Corlu,et al.  Subset selection for simulations accounting for input uncertainty , 2015, 2015 Winter Simulation Conference (WSC).

[73]  Eunhye Song,et al.  Efficient Input Uncertainty Quantification Via Green Simulation Using Sample Path Likelihood Ratios , 2019, 2019 Winter Simulation Conference (WSC).

[74]  Canan G. Corlu,et al.  Demand fulfillment probability under parameter uncertainty , 2016, 2016 Winter Simulation Conference (WSC).

[75]  Liang Ding,et al.  Sequential sampling for Bayesian robust ranking and selection , 2016, 2016 Winter Simulation Conference (WSC).

[76]  Marvin K. Nakayama,et al.  Output Analysis for Simulations , 2006, Proceedings of the 2006 Winter Simulation Conference.

[77]  Barry L. Nelson,et al.  Quickly Assessing Contributions to Input Uncertainty , 2015 .

[78]  Art B. Owen,et al.  Sobol' Indices and Shapley Value , 2014, SIAM/ASA J. Uncertain. Quantification.

[79]  Warren B. Powell,et al.  The Correlated Knowledge Gradient for Simulation Optimization of Continuous Parameters using Gaussian Process Regression , 2011, SIAM J. Optim..

[80]  Wei Xie,et al.  Data-driven stochastic optimization for power grids scheduling under high wind penetration , 2020, Energy Systems.

[81]  Faker Zouaoui,et al.  Accounting for input-model and input-parameter uncertainties in simulation , 2004 .

[82]  Di Wu,et al.  A Bayesian Risk Approach to Data-driven Stochastic Optimization: Formulations and Asymptotics , 2016, SIAM J. Optim..

[83]  Constantine Caramanis,et al.  Theory and Applications of Robust Optimization , 2010, SIAM Rev..

[84]  Wei Xie,et al.  A Bayesian Framework for Quantifying Uncertainty in Stochastic Simulation , 2014, Oper. Res..

[85]  Peter W. Glynn,et al.  CONSTRUCTING SIMULATION OUTPUT INTERVALS UNDER INPUT UNCERTAINTY VIA DATA SECTIONING , 2018, 2018 Winter Simulation Conference (WSC).

[86]  Pu Zhang,et al.  Metamodel-Assisted Sensitivity Analysis for Controlling the Impact of Input Uncertainty , 2019, 2019 Winter Simulation Conference (WSC).

[87]  Yinyu Ye,et al.  Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems , 2010, Oper. Res..

[88]  Wei Xie,et al.  A simulation-based prediction framework for two-stage dynamic decision making , 2016, 2016 Winter Simulation Conference (WSC).

[89]  Henry Lam,et al.  Optimization-based Quantification of Simulation Input Uncertainty via Empirical Likelihood , 2017, 1707.05917.

[90]  Soumyadip Ghosh,et al.  Mirror descent stochastic approximation for computing worst-case stochastic input models , 2015, 2015 Winter Simulation Conference (WSC).

[91]  Anja De Waegenaere,et al.  Robust Solutions of Optimization Problems Affected by Uncertain Probabilities , 2011, Manag. Sci..

[92]  Peter D. Hoff Extending the rank likelihood for semiparametric copula estimation , 2006, math/0610413.

[93]  Enlu Zhou,et al.  ONLINE QUANTIFICATION OF INPUT UNCERTAINTY FOR PARAMETRIC MODELS , 2018, 2018 Winter Simulation Conference (WSC).

[94]  Daniel Kuhn,et al.  Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations , 2015, Mathematical Programming.

[95]  Canan G. Corlu,et al.  Accounting for Parameter Uncertainty in Large-Scale Stochastic Simulations with Correlated Inputs , 2011, Oper. Res..

[96]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[97]  Daniel Kuhn,et al.  Distributionally Robust Convex Optimization , 2014, Oper. Res..

[98]  Enlu Zhou,et al.  Simulation optimization of risk measures with adaptive risk levels , 2016, J. Glob. Optim..

[99]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[100]  Barry L. Nelson,et al.  A quicker assessment of input uncertainty , 2013, 2013 Winter Simulations Conference (WSC).

[101]  Szu Hui Ng,et al.  Reducing input parameter uncertainty for simulations , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[102]  Barry L. Nelson,et al.  Estimating Sensitivity to Input Model Variance , 2019, 2019 Winter Simulation Conference (WSC).

[103]  James R. Wilson,et al.  Accounting for Parameter Uncertainty in Simulation Input Modeling , 2003 .

[104]  Di Wu,et al.  Fixed Confidence Ranking and Selection Under Input Uncertainty , 2019, 2019 Winter Simulation Conference (WSC).

[105]  Warren B. Powell,et al.  The Knowledge-Gradient Policy for Correlated Normal Beliefs , 2009, INFORMS J. Comput..

[106]  R. Cheng,et al.  Two-point methods for assessing variability in simulation output , 1998 .

[107]  Barry L. Nelson,et al.  Input uncertainty and indifference-zone ranking & selection , 2015, 2015 Winter Simulation Conference (WSC).

[108]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[109]  Loo Hay Lee,et al.  Simulation Optimization: A Review and Exploration in the New Era of Cloud Computing and Big Data , 2015, Asia Pac. J. Oper. Res..

[110]  Güzin Bayraksan,et al.  Data-Driven Stochastic Programming Using Phi-Divergences , 2015 .

[111]  Alp Akcay,et al.  Stochastic simulation under input uncertainty for contract-manufacturer selection in pharmaceutical industry , 2016, 2016 Winter Simulation Conference (WSC).

[112]  Shane G. Henderson Input model uncertainty: why do we care and what should we do about it? , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[113]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[114]  Bahar Biller,et al.  Quantifying input uncertainty in an assemble-to-order system simulation with correlated input variables of mixed types , 2014, Proceedings of the Winter Simulation Conference 2014.

[115]  Stephen E. Chick Steps to implement Bayesian input distribution selection , 1999, WSC '99.