Order-based error for managing ensembles of surrogates in mesh adaptive direct search

We investigate surrogate-assisted strategies for global derivative-free optimization using the mesh adaptive direct search (MADS) blackbox optimization algorithm. In particular, we build an ensemble of surrogate models to be used within the search step of MADS to perform global exploration, and examine different methods for selecting the best model for a given problem at hand. To do so, we introduce an order-based error tailored to surrogate-based search. We report computational experiments for ten analytical benchmark problems and three engineering design applications. Results demonstrate that different metrics may result in different model choices and that the use of order-based metrics improves performance.

[1]  A. I. F. Vaz,et al.  Optimizing radial basis functions by d.c. programming and its use in direct search for global derivative-free optimization , 2012 .

[2]  M. Ramu,et al.  Surrogate Based Analysis and Optimization of Roof Slab of Future Fast Breeder Reactor , 2009 .

[3]  Bithin Datta,et al.  Global Optimal Design of Ground Water Monitoring Network Using Embedded Kriging , 2009, Ground water.

[4]  Frances Bauer,et al.  Supercritical Wing Sections III , 1977 .

[5]  A. Messac,et al.  Quantifying Regional Error in Surrogates by Modeling its Relationship with Sample Density , 2013 .

[6]  Benoît Colson,et al.  Trust-region algorithms for derivative-free optimization and nonlinear bilevel programming , 2004, 4OR.

[7]  Thaddeus Tarpey,et al.  A Note on the Prediction Sum of Squares Statistic for Restricted Least Squares , 2000 .

[8]  Charles Audet,et al.  Nonsmooth optimization through Mesh Adaptive Direct Search and Variable Neighborhood Search , 2006, J. Glob. Optim..

[9]  Masoud Rais-Rohani,et al.  Ensemble of Metamodels with Optimized Weight Factors , 2008 .

[10]  L. Shawn Matott,et al.  Application of MATLAB and Python optimizers to two case studies involving groundwater flow and contaminant transport modeling , 2011, Comput. Geosci..

[11]  Charles Audet,et al.  A Survey on Direct Search Methods for Blackbox Optimization and their Applications , 2014 .

[12]  R. Haftka,et al.  Ensemble of surrogates , 2007 .

[13]  Stefan M. Wild,et al.  BENEFITS OF DEEPER ANALYSIS IN SIMULATION- BASED GROUNDWATER OPTIMIZATION PROBLEMS , 2012 .

[14]  Ilan Kroo,et al.  Aircraft Design: Synthesis and Analysis , 1999 .

[15]  F. Bauer,et al.  Supercritical wing sections. III. [codes K and corrections to H, in FORTRAN IV for CDC 6600] , 1977 .

[16]  A. Messac,et al.  Predictive quantification of surrogate model fidelity based on modal variations with sample density , 2015 .

[17]  Robert Piché,et al.  Mixture surrogate models based on Dempster-Shafer theory for global optimization problems , 2011, J. Glob. Optim..

[18]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[19]  Sébastien Le Digabel,et al.  Use of quadratic models with mesh-adaptive direct search for constrained black box optimization , 2011, Optim. Methods Softw..

[20]  Rommel G. Regis,et al.  Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions , 2011, Comput. Oper. Res..

[21]  James R. Craig,et al.  Pump-and-treat optimization using analytic element method flow models , 2006 .

[22]  J. Mark.  Introduction to radial basis function networks , 1996 .

[23]  Jacobus E. Rooda,et al.  A classification of methods for distributed system optimization based on formulation structure , 2009 .

[24]  Klaus Schittkowski,et al.  More test examples for nonlinear programming codes , 1981 .

[25]  Charles Audet,et al.  A surrogate-model-based method for constrained optimization , 2000 .

[26]  P. Toint,et al.  An Algorithm using Quadratic Interpolation for Unconstrained Derivative Free Optimization , 1996 .

[27]  J. D Gergonne,et al.  The application of the method of least squares to the interpolation of sequences , 1974 .

[28]  R. Haftka,et al.  Multiple Surrogates for the Shape Optimization of Bluff Body-Facilitated Mixing , 2005 .

[29]  Jean-François Petiot,et al.  Brass Instruments Design Using Physics-Based Sound Simulation Models and Surrogate-Assisted Derivative-Free Optimization , 2016, DAC 2016.

[30]  Layne T. Watson,et al.  Efficient global optimization algorithm assisted by multiple surrogate techniques , 2012, Journal of Global Optimization.

[31]  S. Stigler.  Gergonne's 1815 paper on the design and analysis of polynomial regression experiments , 1974 .

[32]  Raphael T. Haftka,et al.  Surrogate-based Analysis and Optimization , 2005 .

[33]  Martin D. Buhmann,et al.  Radial Basis Functions: Theory and Implementations: Preface , 2003 .

[34]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[35]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[36]  Charles Audet,et al.  A MADS Algorithm with a Progressive Barrier for Derivative-Free Nonlinear Programming , 2007 .

[37]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[38]  B. Eaves.  On Quadratic Programming , 1971 .

[39]  Robert B. Gramacy,et al.  Dynamic Trees for Learning and Design , 2009, 0912.1586.

[40]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[41]  E. Hines,et al.  Combining Gaussian processes, mutual information and a genetic algorithm for multi-target optimization of expensive-to-evaluate functions , 2014 .

[42]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[43]  Christine A. Shoemaker,et al.  Influence of ensemble surrogate models and sampling strategy on the solution quality of algorithms for computationally expensive black-box global optimization problems , 2014, J. Glob. Optim..

[44]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[45]  Sébastien Le Digabel,et al.  The mesh adaptive direct search algorithm with treed Gaussian process surrogates , 2011 .

[46]  G. S. Watson.  Smoothing and interpolation by kriging and with splines , 1984 .

[47]  Yoel Tenne,et al.  An adaptive-topology ensemble algorithm for engineering optimization problems , 2015 .

[48]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[49]  Willi Hock,et al.  Lecture Notes in Economics and Mathematical Systems , 1981 .

[50]  Sébastien Le Digabel,et al.  Algorithm xxx : NOMAD : Nonlinear Optimization with the MADS algorithm , 2010 .

[51]  Valder Steffen,et al.  Ensemble of Surrogates: a Framework based on Minimization of the Mean Integrated Square Error , 2008 .

[52]  Srinivas Kodiyalam.  Multidisciplinary Aerospace Systems Optimization---Computational AeroSciences (CAS) Project , 2001 .

[53]  Hui-Chung Yeh,et al.  Rainfall network design using kriging and entropy , 2008 .

[54]  R. Regis.  Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points , 2014 .

[55]  Sébastien Le Digabel,et al.  Statistical Surrogate Formulations for Simulation-Based Design Optimization , 2015 .

[56]  N. M. Alexandrov,et al.  A trust-region framework for managing the use of approximation models in optimization , 1997 .

[57]  Christodoulos A. Floudas,et al.  ARGONAUT: AlgoRithms for Global Optimization of coNstrAined grey-box compUTational problems , 2017, Optim. Lett..

[58]  Christine A. Shoemaker,et al.  ORBIT: Optimization by Radial Basis Function Interpolation in Trust-Regions , 2008, SIAM J. Sci. Comput..

[59]  Nielen Stander,et al.  Comparing three error criteria for selecting radial basis function network topology , 2009 .

[60]  Jean-François Dubé,et al.  Decomposition of multidisciplinary optimization problems: formulations and application to a simplified wing design , 2005 .

[61]  Nicholas I. M. Gould,et al.  Global Convergence of a Trust-Region SQP-Filter Algorithm for General Nonlinear Programming , 2002, SIAM J. Optim..

[62]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[63]  Bernhard Sendhoff,et al.  A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation , 2007, GECCO '07.

[64]  K. Yamazaki,et al.  Sequential Approximate Optimization using Radial Basis Function network for engineering optimization , 2011 .

[65]  Christodoulos A. Floudas,et al.  Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption , 2017, J. Glob. Optim..

[66]  Jian Liu,et al.  An efficient ensemble of radial basis functions method based on quadratic programming , 2016 .

[67]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[68]  Charles Audet,et al.  Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems , 2008 .

[69]  Sébastien Le Digabel,et al.  Modeling an Augmented Lagrangian for Blackbox Constrained Optimization , 2014, Technometrics.

[70]  Jaroslaw Sobieszczanski-Sobieski,et al.  Supersonic Business Jet Design Through Bi-Level Integrated System Synthesis , 1999 .

[71]  Humberto Rocha,et al.  Incorporating minimum Frobenius norm models in direct search , 2010, Comput. Optim. Appl..

[72]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[73]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[74]  Charles Audet,et al.  Mesh Adaptive Direct Search Algorithms for Constrained Optimization , 2006, SIAM J. Optim..

[75]  Pierre Hansen,et al.  Variable neighborhood search , 1997, Eur. J. Oper. Res..

[76]  Luís N. Vicente,et al.  A particle swarm pattern search method for bound constrained global optimization , 2007, J. Glob. Optim..

[77]  Charles Audet,et al.  OrthoMADS: A Deterministic MADS Instance with Orthogonal Directions , 2008, SIAM J. Optim..