Learning surrogate models for simulation‐based optimization

A central problem in modeling, namely that of learning an algebraic model from data obtained from simulations or experiments is addressed. A methodology that uses a small number of simulations or experiments to learn models that are as accurate and as simple as possible is proposed. The approach begins by building a low-complexity surrogate model. The model is built using a best subset technique that leverages an integer programming formulation to allow for the efficient consideration of a large number of possible functional components in the model. The model is then improved systematically through the use of derivative-free optimization solvers to adaptively sample new simulation or experimental points. Automated learning of algebraic models for optimization (ALAMO), the computational implementation of the proposed methodology, along with examples and extensive computational comparisons between ALAMO and a variety of machine learning techniques, including Latin hypercube sampling, simple least-squares regression, and the lasso is described. © 2014 American Institute of Chemical Engineers AIChE J, 60: 2211–2227, 2014

[1]  D. Krige.  A statistical approach to some basic mine valuation problems on the Witwatersrand, by D.G. Krige, published in the Journal, December 1951 : introduction by the author , 1951 .

[2]  Roger G.E. Franks,et al.  Modeling and Simulation in Chemical Engineering , 1972 .

[3]  H. Akaike.  A new look at the statistical model identification , 1974 .

[4]  J. Gathen,et al.  A bound on solutions of linear integer equalities and inequalities , 1978 .

[5]  Babu Joseph,et al.  ASPEN: An Advanced System for Process Engineering , 1979 .

[6]  Alexander P. Morgan,et al.  A Method for Computing All Solutions to Systems of Polynomials Equations , 1983, TOMS.

[7]  Constantinos C. Pantelides,et al.  SpeedUp—recent advances in process simulation , 1988 .

[8]  N. Cressie.  Spatial prediction and ordinary kriging , 1988 .

[9]  F. Pukelsheim.  Optimal Design of Experiments , 1993 .

[10]  Clifford M. Hurvich,et al.  A CORRECTED AKAIKE INFORMATION CRITERION FOR VECTOR AUTOREGRESSIVE MODEL SELECTION , 1993 .

[11]  M.H. Hassoun,et al.  Fundamentals of Artificial Neural Networks , 1996, Proceedings of the IEEE.

[12]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[13]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[14]  Tom Dhaene,et al.  Adaptive CAD-model building algorithm for general planar microwave structures , 1999 .

[15]  D. Sorensen,et al.  A Survey of Model Reduction Methods for Large-Scale Systems , 2000 .

[16]  Timothy W. Simpson,et al.  Metamodels for Computer-based Engineering Design: Survey and recommendations , 2001, Engineering with Computers.

[17]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[18]  Michael C. Fu,et al.  Feature Article: Optimization for simulation: Theory vs. Practice , 2002, INFORMS J. Comput..

[19]  Matthew J. Realff,et al.  Metamodeling Approach to Optimization of Steady-State Flowsheet Simulations: Model Generation , 2002 .

[20]  Michael C. Fu,et al.  Optimization for Simulation: Theory vs. Practice , 2002 .

[21]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[22]  Dennis K. J. Lin,et al.  Ch. 4. Uniform experimental designs and their applications in industry , 2003 .

[23]  Fred Glover,et al.  Practical introduction to simulation optimization , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[24]  Erricos John Kontoghiorghes,et al.  Parallel algorithms for computing all possible subset regression models using the QR decomposition , 2003, Parallel Comput..

[25]  Y. Selen,et al.  Model-order selection: a review of information criterion rules , 2004, IEEE Signal Processing Magazine.

[26]  Fred W. Glover,et al.  Simulation optimization: a review, new developments, and applications , 2005, Proceedings of the Winter Simulation Conference, 2005..

[27]  Nikolaos V. Sahinidis,et al.  A polyhedral branch-and-cut approach to global optimization , 2005, Math. Program..

[28]  N. Zheng,et al.  Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models , 2006, J. Glob. Optim..

[29]  Lorenz T. Biegler,et al.  On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming , 2006, Math. Program..

[30]  G. Gary Wang,et al.  Review of Metamodeling Techniques in Support of Engineering Design Optimization , 2007, DAC 2006.

[31]  M. Ierapetritou,et al.  A kriging method for the solution of nonlinear programs with black‐box functions , 2007 .

[32]  Erricos John Kontoghiorghes,et al.  A graph approach to generate all possible regression submodels , 2007, Comput. Stat. Data Anal..

[33]  Arnold Neumaier,et al.  SNOBFIT -- Stable Noisy Optimization by Branch and Fit , 2008, TOMS.

[34]  Katya Scheinberg,et al.  Geometry of interpolation sets in derivative free optimization , 2007, Math. Program..

[35]  I. Grossmann,et al.  An algorithm for the use of surrogate models in modular flowsheet optimization , 2008 .

[36]  Dirk Gorissen,et al.  A novel sequential design strategy for global surrogate modeling , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[37]  Hod Lipson,et al.  Distilling Free-Form Natural Laws from Experimental Data , 2009, Science.

[38]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[39]  Weifeng Chen,et al.  Interfacing IPOPT with Aspen Open Solvers and CAPE-OPEN , 2009 .

[40]  Piet Demeester,et al.  A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design , 2010, J. Mach. Learn. Res..

[41]  Tom Dhaene,et al.  Efficient space-filling and non-collapsing sequential design strategies for simulation-based modeling , 2011, Eur. J. Oper. Res..

[42]  Christos T. Maravelias,et al.  Surrogate‐based superstructure optimization framework , 2011 .

[43]  David C. Miller,et al.  A One-Dimensional (1-D) Three-Region Model for a Bubbling Fluidized-Bed Adsorber , 2012 .

[44]  Nikolaos V. Sahinidis,et al.  Derivative-free optimization: a review of algorithms and comparison of software implementations , 2013, J. Glob. Optim..

[45]  Michael A. Saunders,et al.  User's Guide for SNOPT Version 7.4: Software for Large-Scale Nonlinear Programming , 2015 .