METAMODELING A SYSTEM DYNAMICS MODEL : A CONTEMPORARY COMPARISON OF METHODS

Advancements in computer technology have resulted in improved computing capability and software functionality. Concurrently, in the simulation community demand to study complex, integrated systems has grown. As a result, it is difficult to perform model exploration or optimization simply due to time and resource limitations. Metamodeling offers an approach to overcome this issue; however, limited study has been made to compare the methods most appropriate for simulation modeling. This paper presents a contemporary comparison of methods useful for creating a metamodel of a simulation model. For comparison we explore the performance of a complex system dynamics model of a community hospital. In our view several characteristics of hospital operations present an interesting challenge to explore and compare the well-known competing methods. We consider three dimensions in our comparison: fit quality, fitting time, and results interpretability. The paper discusses the better performing methods corresponding to these dimensions and considers tradeoffs.

[1]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, WSC 2008.

[2]  J. Friedman.  Stochastic gradient boosting , 2002 .

[3]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[4]  A. Gelfand,et al.  Handbook of spatial statistics , 2010 .

[5]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[6]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[7]  Chen-Fu Chien,et al.  Efficient development of cycle time response surfaces using progressive simulation metamodeling , 2014 .

[8]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[9]  Franccois Bachoc,et al.  Calibration and Improved Prediction of Computer Models by Universal Kriging , 2013, 1301.4114.

[10]  Julian D. Olden,et al.  Illuminating the “black box”: a randomization approach for understanding variable contributions in artificial neural networks , 2002 .

[11]  Barry L. Nelson,et al.  Moving Least Squares regression for high dimensional simulation metamodeling , 2012, Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC).

[12]  Ying Hung,et al.  PENALIZED BLIND KRIGING IN COMPUTER EXPERIMENTS , 2011 .

[13]  Xi Chen.  Enhancing Stochastic Kriging Metamodels for Computer Simulation , 2012 .

[14]  G. Gary Wang,et al.  Review of Metamodeling Techniques in Support of Engineering Design Optimization , 2007, DAC 2006.

[15]  Raymond L. Smith,et al.  Sensitivity analysis for a whole hospital system dynamics model , 2014, Proceedings of the Winter Simulation Conference 2014.

[16]  G. Box,et al.  Introduction to Response Surface Methodology , 2006 .

[17]  Jeremy Staum,et al.  Better simulation metamodeling: The why, what, and how of stochastic kriging , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[18]  Hans-Peter Piepho,et al.  A comparison of random forests, boosting and support vector machines for genomic selection , 2011, BMC proceedings.

[19]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[20]  Jack P. C. Kleijnen,et al.  Kriging Metamodeling in Simulation: A Review , 2007, Eur. J. Oper. Res..

[21]  Trevor J. Ringrose,et al.  GAMLSS and neural networks in combat simulation metamodelling: A case study , 2013, Expert Syst. Appl..

[22]  Gabriella Dellino,et al.  Robust simulation optimization methods using Kriging metamodels , 2009 .

[23]  Yunqian Ma,et al.  Practical selection of SVM parameters and noise estimation for SVM regression , 2004, Neural Networks.

[24]  G. Box,et al.  On the Experimental Attainment of Optimum Conditions , 1951 .

[25]  Jack P. C. Kleijnen,et al.  Kriging metamodeling in constrained simulation optimization: an explorative study , 2007, 2007 Winter Simulation Conference.

[26]  Jack P. C. Kleijnen,et al.  Stochastic Intrinsic Kriging for Simulation Metamodelling , 2015 .