The use of search experimentation in discrete-event simulation practice

The practice of DES experimentation has not been rigorously assessed in over a decade. Past studies of DES practice report little transfer of experimentation theory into real-world application. We conducted an international survey of over 300 modellers to investigate the extent to which simulation optimisation, meta-modelling and design of experiments are used in practice. Over the last decade there has been substantial growth in the use of optimisation and to a lesser extent design of experiments to tackle practical problems. However, users rarely make use of optimisers bundled with commercial software, opting instead for custom or third-party solutions. Outside of academia, the use of methods is hampered by a lack of application knowledge and a persisting view that such techniques are not necessary. It is clear that academics must not become complacent regarding the dissemination of theory into common practice and continue to reach out to industry users.

[1]  Dan Muller Automod™ - providing simulation solutions for over 25 years , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[2]  Hugh J. Watson,et al.  The Application of Simulation: A Survey of Industry Practice , 1983 .

[3]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, WSC 2008.

[4]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[5]  Jack P. C. Kleijnen,et al.  Simulation experiments in practice: statistical design and regression analysis , 2007, J. Simulation.

[6]  Brian W. Hollocks,et al.  Discrete-event simulation: an inquiry into user practice , 2001, Simul. Pract. Theory.

[7]  Stewart Robinson,et al.  Discrete-event simulation: from the pioneers to the present, what next? , 2005, J. Oper. Res. Soc..

[8]  Sally C. Brailsford,et al.  Using simulation to improve the blood supply chain , 2007, J. Oper. Res. Soc..

[9]  Jack P. C. Kleijnen,et al.  An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis , 2005, Eur. J. Oper. Res..

[10]  Michael Pidd,et al.  Simulation modelling is 50! Do we need a reality check? , 2009, J. Oper. Res. Soc..

[11]  Enver Yücesan,et al.  Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey , 2003, TOMC.

[12]  A. R. Zinsmeister,et al.  Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building, by G. E. P. Box, W. G. Hunter, and J. S. Hunter , 1981 .

[13]  Stewart Robinson,et al.  Conceptual modelling for simulation Part II: a framework for conceptual modelling , 2008, J. Oper. Res. Soc..

[14]  Jack P. C. Kleijnen,et al.  Constrained optimization in expensive simulation: Novel approach , 2010, Eur. J. Oper. Res..

[15]  Jack P. C. Kleijnen,et al.  Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations , 2009, J. Oper. Res. Soc..

[16]  Averill M. Law,et al.  Simulation modelling and analysis , 1991 .

[17]  Jack P. C. Kleijnen,et al.  Kriging Metamodeling in Simulation: A Review , 2007, Eur. J. Oper. Res..

[18]  Michael Pidd,et al.  DGHPSIM:: Generic simulation of hospital performance , 2011, TOMC.

[19]  Michael C. Fu,et al.  Optimization for Simulation: Theory vs. Practice , 2002 .

[20]  J. S. Hunter,et al.  Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. , 1979 .

[21]  Robert Fildes,et al.  The foundation, development and current practice of OR: An editorial introduction and overview , 1998, J. Oper. Res. Soc..

[22]  Kuo-Hao Chang,et al.  Stochastic Nelder-Mead simplex method - A new globally convergent direct search method for simulation optimization , 2012, Eur. J. Oper. Res..

[23]  Michael Pidd,et al.  Tools for Thinking—Modelling in Management Science , 1997 .

[24]  Stewart Robinson,et al.  Conceptual modelling for simulation Part I: definition and requirements , 2008, J. Oper. Res. Soc..

[25]  Thomas Monks,et al.  A note on the use of multiple comparison scenario techniques in education and practice , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[26]  Hamidreza Eskandari,et al.  Performance analysis of comercial simulation-based optimization packages: OptQuest and Witness Optimizer , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[27]  Barry L. Nelson,et al.  Discrete Optimization via Simulation Using COMPASS , 2006, Oper. Res..

[28]  Michael C. Fu,et al.  Feature Article: Optimization for simulation: Theory vs. Practice , 2002, INFORMS J. Comput..

[29]  Ruth Davies,et al.  AutoSimOA: a framework for automated analysis of simulation output , 2011, J. Simulation.

[30]  Fred W. Glover,et al.  Simulation optimization: a review, new developments, and applications , 2005, Proceedings of the Winter Simulation Conference, 2005..

[31]  Xi Chen,et al.  Enhancing Stochastic Kriging Metamodels with Gradient Estimators , 2013, Oper. Res..

[32]  Soemon Takakuwa,et al.  Optimization and analysis of staffing problems at a retail store , 2010, Proceedings of the 2010 Winter Simulation Conference.

[33]  David T. Sturrock,et al.  Recent innovations in Simio , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[34]  Abhijit Gosavi,et al.  Simulation-based optimisation for material dispatching in Vendor-Managed Inventory systems , 2007, Int. J. Simul. Process. Model..

[35]  Barry L. Nelson,et al.  Recent advances in ranking and selection , 2007, 2007 Winter Simulation Conference.

[36]  Thomas W. Lucas,et al.  Designs for Large‐Scale Simulation Experiments, with Applications to Defense and Homeland Security , 2012 .

[37]  Jack P. C. Kleijnen,et al.  A User's Guide to the Brave New World of Designing Simulation Experiments. State-of-the-Art Review , 2005 .

[38]  Reid Kress,et al.  Discrete event simulation class for engineering graduate students , 2010, Proceedings of the 2010 Winter Simulation Conference.

[39]  Mamadou D. Seck,et al.  The simulation-based multi-objective evolutionary optimization (SIMEON) framework , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[40]  Fred Glover,et al.  Practical introduction to simulation optimization , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[41]  Celso Leandro Palma,et al.  Simulation: The Practice of Model Development and Use , 2016 .

[42]  Linda Weiser Friedman,et al.  The Metamodel in Simulation Analysis: Can It Be Trusted? , 1988 .

[43]  Luke D. Postema,et al.  The Institute of Electrical and Electronics Engineers , 1963, Nature.

[44]  Ali Ajdari,et al.  An Adaptive Exploration-Exploitation Algorithm for Constructing Metamodels in Random Simulation Using a Novel Sequential Experimental Design , 2014, Commun. Stat. Simul. Comput..

[45]  Jack P. C. Kleijnen,et al.  Validation of regression metamodels in simulation: Bootstrap approach , 2006, Eur. J. Oper. Res..

[46]  Valerie Belton,et al.  Decision support systems: Learning from visual interactive modelling , 1994, Decis. Support Syst..

[47]  Enrique del Castillo,et al.  Statitical Testing of Optimality Conditions in Multiresponse Simulation-Based Optimization , 2005, Eur. J. Oper. Res..

[48]  Jie Xu,et al.  An Adaptive Hyperbox Algorithm for High-Dimensional Discrete Optimization via Simulation Problems , 2013, INFORMS J. Comput..

[49]  Kai Virtanen,et al.  Simulation metamodeling with dynamic Bayesian networks , 2011, Eur. J. Oper. Res..