Order-based error for managing ensembles of surrogates in derivative-free optimization

We investigate surrogate-assisted strategies for derivative-free optimization using the mesh adaptive direct search (MADS) blackbox optimization algorithm. In particular, we build an ensemble of surrogate models to be used within the search step of MADS, and examine different methods for selecting the best model for a given problem at hand. To do so, we introduce an order-based error tailored to surrogate-based search. We report computational experiments for ten analytical benchmark problems and two engineering design applications. Results demonstrate that different metrics may result in different model choices and that the use of order-based metrics improves performance.

[1]  Sébastien Le Digabel,et al.  Algorithm xxx : NOMAD : Nonlinear Optimization with the MADS algorithm , 2010 .

[2]  Thaddeus Tarpey,et al.  A Note on the Prediction Sum of Squares Statistic for Restricted Least Squares , 2000 .

[3]  Masoud Rais-Rohani,et al.  Ensemble of Metamodels with Optimized Weight Factors , 2008 .

[4]  Charles Audet,et al.  Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems , 2008 .

[5]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[6]  E. Hines,et al.  Combining Gaussian processes, mutual information and a genetic algorithm for multi-target optimization of expensive-to-evaluate functions , 2014 .

[7]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[8]  Sébastien Le Digabel,et al.  The mesh adaptive direct search algorithm with treed Gaussian process surrogates , 2011 .

[9]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[10]  Charles Audet,et al.  Nonsmooth optimization through Mesh Adaptive Direct Search and Variable Neighborhood Search , 2006, J. Glob. Optim..

[11]  Valder Steffen,et al.  Ensemble of Surrogates: a Framework based on Minimization of the Mean Integrated Square Error , 2008 .

[12]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[13]  Charles Audet,et al.  A surrogate-model-based method for constrained optimization , 2000 .

[14]  Bithin Datta,et al.  Global Optimal Design of Ground Water Monitoring Network Using Embedded Kriging , 2009, Ground water.

[15]  J. Mark.  Introduction to radial basis function networks , 1996 .

[16]  N. M. Alexandrov,et al.  A trust-region framework for managing the use of approximation models in optimization , 1997 .

[17]  Willi Hock,et al.  Lecture Notes in Economics and Mathematical Systems , 1981 .

[18]  Nielen Stander,et al.  Comparing three error criteria for selecting radial basis function network topology , 2009 .

[19]  Yoel Tenne,et al.  An adaptive-topology ensemble algorithm for engineering optimization problems , 2015 .

[20]  Jean-François Dubé,et al.  Decomposition of multidisciplinary optimization problems: formulations and application to a simplified wing design , 2005 .

[21]  Sébastien Le Digabel,et al.  Statistical Surrogate Formulations for Simulation-Based Design Optimization , 2015 .

[22]  Charles Audet,et al.  Mesh Adaptive Direct Search Algorithms for Constrained Optimization , 2006, SIAM J. Optim..

[23]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[24]  Charles Audet,et al.  A MADS Algorithm with a Progressive Barrier for Derivative-Free Nonlinear Programming , 2007 .

[25]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[26]  Raphael T. Haftka,et al.  Surrogate-based Analysis and Optimization , 2005 .

[27]  Robert B. Gramacy,et al.  Dynamic Trees for Learning and Design , 2009, 0912.1586.

[28]  S. Stigler.  Gergonne's 1815 paper on the design and analysis of polynomial regression experiments , 1974 .

[29]  J. D Gergonne,et al.  The application of the method of least squares to the interpolation of sequences , 1974 .

[30]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[31]  Srinivas Kodiyalam.  Multidisciplinary Aerospace Systems Optimization---Computational AeroSciences (CAS) Project , 2001 .

[32]  Robert Piché,et al.  Mixture surrogate models based on Dempster-Shafer theory for global optimization problems , 2011, J. Glob. Optim..

[33]  Sébastien Le Digabel,et al.  Use of quadratic models with mesh-adaptive direct search for constrained black box optimization , 2011, Optim. Methods Softw..

[34]  Martin D. Buhmann,et al.  Radial Basis Functions: Theory and Implementations: Preface , 2003 .

[35]  Benoît Colson,et al.  Trust-region algorithms for derivative-free optimization and nonlinear bilevel programming , 2004, 4OR.

[36]  Hui-Chung Yeh,et al.  Rainfall network design using kriging and entropy , 2008 .

[37]  Luís N. Vicente,et al.  A particle swarm pattern search method for bound constrained global optimization , 2007, J. Glob. Optim..

[38]  A. Messac,et al.  Quantifying Regional Error in Surrogates by Modeling its Relationship with Sample Density , 2013 .

[39]  Bernhard Sendhoff,et al.  A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation , 2007, GECCO '07.

[40]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[41]  Humberto Rocha,et al.  Incorporating minimum Frobenius norm models in direct search , 2010, Comput. Optim. Appl..

[42]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[43]  Charles Audet,et al.  A Survey on Direct Search Methods for Blackbox Optimization and their Applications , 2014 .

[44]  B. Eaves.  On Quadratic Programming , 1971 .

[45]  Charles Audet,et al.  Blackbox and derivative-free optimization: theory, algorithms and applications , 2016 .

[46]  G. S. Watson.  Smoothing and interpolation by kriging and with splines , 1984 .

[47]  A. Messac,et al.  Predictive quantification of surrogate model fidelity based on modal variations with sample density , 2015 .

[48]  Klaus Schittkowski,et al.  More test examples for nonlinear programming codes , 1981 .

[49]  Christine A. Shoemaker,et al.  ORBIT: Optimization by Radial Basis Function Interpolation in Trust-Regions , 2008, SIAM J. Sci. Comput..

[50]  P. Toint,et al.  An Algorithm using Quadratic Interpolation for Unconstrained Derivative Free Optimization , 1996 .

[51]  A. I. F. Vaz,et al.  Optimizing radial basis functions by d.c. programming and its use in direct search for global derivative-free optimization , 2012 .

[52]  Layne T. Watson,et al.  Efficient global optimization algorithm assisted by multiple surrogate techniques , 2012, Journal of Global Optimization.

[53]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[54]  R. Haftka,et al.  Ensemble of surrogates , 2007 .

[55]  R. Haftka,et al.  Multiple Surrogates for the Shape Optimization of Bluff Body-Facilitated Mixing , 2005 .

[56]  Nicholas I. M. Gould,et al.  Global Convergence of a Trust-Region SQP-Filter Algorithm for General Nonlinear Programming , 2002, SIAM J. Optim..

[57]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[58]  Jian Liu,et al.  An efficient ensemble of radial basis functions method based on quadratic programming , 2016 .

[59]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[60]  K. Yamazaki,et al.  Sequential Approximate Optimization using Radial Basis Function network for engineering optimization , 2011 .

[61]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[62]  Sébastien Le Digabel.  NOMAD: Nonlinear Optimization with the MADS Algorithm , 2009 .