Optimization of black-box problems using Smolyak grids and polynomial approximations

A surrogate-based optimization method is presented, which aims to locate the global optimum of box-constrained problems using input–output data. The method starts with a global search of the n-dimensional space, using a Smolyak (Sparse) grid which is constructed using Chebyshev extrema in the one-dimensional space. The collected samples are used to fit polynomial interpolants, which are used as surrogates towards the search for the global optimum. The proposed algorithm adaptively refines the grid by collecting new points in promising regions, and iteratively refines the search space around the incumbent sample until the search domain reaches a minimum hyper-volume and convergence has been attained. The algorithm is tested on a large set of benchmark problems with up to thirty dimensions and its performance is compared to a recent algorithm for global optimization of grey-box problems using quadratic, kriging and radial basis functions. It is shown that the proposed algorithm has a consistently reliable performance for the vast majority of test problems, and this is attributed to the use of Chebyshev-based Sparse Grids and polynomial interpolants, which have not gained significant attention in surrogate-based optimization thus far.

[1]  P. N. Suganthan,et al.  Differential Evolution: A Survey of the State-of-the-Art , 2011, IEEE Transactions on Evolutionary Computation.

[2]  Brendan Harding,et al.  Adaptive Sparse Grids and Extrapolation Techniques , 2016 .

[3]  Tapan Mukerji,et al.  Derivative-Free Optimization for Oil Field Operations , 2011, Computational Optimization and Applications in Engineering and Industry.

[4]  I. Grossmann,et al.  An algorithm for the use of surrogate models in modular flowsheet optimization , 2008 .

[5]  Bjarne Grimstad,et al.  Global optimization with spline constraints: a new branch-and-bound method based on B-splines , 2016, J. Glob. Optim..

[6]  L. Trefethen.  Six myths of polynomial interpolation and quadrature , 2011 .

[7]  Aimo A. Törn,et al.  Global Optimization , 1999, Science.

[8]  Marianthi G. Ierapetritou,et al.  A Kriging-Based Approach to MINLP Containing Black-Box Models and Noise , 2008 .

[9]  Christodoulos A. Floudas,et al.  ARGONAUT: AlgoRithms for Global Optimization of coNstrAined grey-box compUTational problems , 2017, Optim. Lett..

[10]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[11]  Nikolaos V. Sahinidis,et al.  Derivative-free optimization: a review of algorithms and comparison of software implementations , 2013, J. Glob. Optim..

[12]  Christos T. Maravelias,et al.  Surrogate-Based Process Synthesis , 2010 .

[13]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[14]  Dirk Pflüger,et al.  Adaptive Sparse Grid Techniques for Data Mining , 2006, HPSC.

[15]  Grzegorz W. Wasilkowski,et al.  Smolyak's Algorithm for Integration and L1-Approximation of Multivariate Functions with Bounded Mixed Derivatives of Second Order , 2004, Numerical Algorithms.

[16]  Lilia Maliar,et al.  Smolyak Method for Solving Dynamic Economic Models: Lagrange Interpolation, Anisotropic Grid and Adaptive Domain , 2013 .

[17]  David C. Miller,et al.  Learning surrogate models for simulation‐based optimization , 2014 .

[18]  Christodoulos A. Floudas,et al.  Global optimization advances in Mixed-Integer Nonlinear Programming, MINLP, and Constrained Derivative-Free Optimization, CDFO , 2016, Eur. J. Oper. Res..

[19]  Tamara G. Kolda,et al.  Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods , 2003, SIAM Rev..

[20]  Erich Novak,et al.  Global Optimization Using Hyperbolic Cross Points , 1996 .

[21]  Edoardo Amaldi,et al.  PGS-COM: A hybrid method for constrained non-smooth black-box optimization problems: Brief review, novel algorithm and comparative evaluation , 2014, Comput. Chem. Eng..

[22]  Lorenz T. Biegler,et al.  A trust region filter method for glass box/black box optimization , 2016 .

[23]  Marianthi G. Ierapetritou,et al.  Surrogate-Based Optimization of Expensive Flowsheet Modeling for Continuous Pharmaceutical Manufacturing , 2013, Journal of Pharmaceutical Innovation.

[24]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[25]  Dinh DźNg.  Sampling and Cubature on Sparse Grids Based on a B-spline Quasi-Interpolation , 2016 .

[26]  Robert Hooke,et al.  `` Direct Search'' Solution of Numerical and Statistical Problems , 1961, JACM.

[27]  Hans-Joachim Bungartz,et al.  Higher Order Quadrature on Sparse Grids , 2004, International Conference on Computational Science.

[28]  Julio R. Banga,et al.  Scatter search for chemical and bio-process optimization , 2007, J. Glob. Optim..

[29]  A. I. F. Vaz,et al.  Optimizing radial basis functions by d.c. programming and its use in direct search for global derivative-free optimization , 2012 .

[30]  Sébastien Le Digabel,et al.  Use of quadratic models with mesh-adaptive direct search for constrained black box optimization , 2011, Optim. Methods Softw..

[31]  Thomas Gerstner,et al.  Numerical integration using sparse grids , 2004, Numerical Algorithms.

[32]  Marianthi G. Ierapetritou,et al.  Derivative‐free optimization for expensive constrained problems using a novel expected improvement objective function , 2014 .

[33]  Dirk Reith,et al.  SpaGrOW - A Derivative-Free Optimization Scheme for Intermolecular Force Field Parameters Based on Sparse Grid Methods , 2013, Entropy.

[34]  Xiaojun Chen,et al.  Error bounds for approximation in Chebyshev points , 2010, Numerische Mathematik.

[35]  Nikolaos V. Sahinidis,et al.  The ALAMO approach to machine learning , 2017, Comput. Chem. Eng..

[36]  Alfio Quarteroni,et al.  A new algorithm for high-dimensional uncertainty quantification based on dimension-adaptive sparse grid approximation and reduced basis methods , 2015, J. Comput. Phys..

[37]  Hans-Joachim Bungartz,et al.  Multivariate Quadrature on Adaptive Sparse Grids , 2003, Computing.

[38]  M. Patriksson,et al.  A method for simulation based optimization using radial basis functions , 2010 .

[39]  Sethuraman Sankaran,et al.  Stochastic optimization using a sparse grid collocation scheme , 2009 .

[40]  Piotr Gajda,et al.  Smolyak's algorithm for weighted L1-approximation of multivariate functions with bounded rth mixed derivatives over ℝd , 2005, Numerical Algorithms.

[41]  Bengt Fornberg,et al.  The Runge phenomenon and spatially variable shape parameters in RBF interpolation , 2007, Comput. Math. Appl..

[42]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[43]  A. Keane,et al.  Evolutionary Optimization of Computationally Expensive Problems via Surrogate Modeling , 2003 .

[44]  Hans-Martin Gutmann,et al.  A Radial Basis Function Method for Global Optimization , 2001, J. Glob. Optim..

[45]  Julio R. Banga,et al.  An evolutionary method for complex-process optimization , 2010, Comput. Oper. Res..

[46]  Matthew J. Realff,et al.  Optimization and Validation of Steady-State Flowsheet Simulation Metamodels , 2002 .

[47]  Benjamin Peherstorfer,et al.  Spatially adaptive sparse grids for high-dimensional data-driven problems , 2010, J. Complex..

[48]  Nikolaos V. Sahinidis,et al.  Simulation optimization: a review of algorithms and applications , 2014, 4OR.

[49]  R. Regis.  Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points , 2014 .

[50]  Guiqiao Xu,et al.  On weak tractability of the Smolyak algorithm for approximation problems , 2015, J. Approx. Theory.

[51]  Benjamin Peherstorfer,et al.  Selected Recent Applications of Sparse Grids , 2015 .

[52]  W. Sickel,et al.  Spline interpolation on sparse grids , 2011 .

[53]  M. Powell.  The BOBYQA algorithm for bound constrained optimization without derivatives , 2009 .

[54]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[55]  Selen Cremaschi,et al.  Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models , 2012, Comput. Chem. Eng..

[56]  Marianthi G. Ierapetritou,et al.  A kriging based method for the solution of mixed-integer nonlinear programs containing black-box functions , 2009, J. Glob. Optim..

[57]  M. Powell.  A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation , 1994 .

[58]  Christine A. Shoemaker,et al.  SO-MI: A surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems , 2013, Comput. Oper. Res..

[59]  Henryk Wozniakowski,et al.  Explicit Cost Bounds of Algorithms for Multivariate Tensor Product Problems , 1995, J. Complex..

[60]  Philip Rabinowitz,et al.  Methods of Numerical Integration , 1985 .

[61]  Christodoulos A. Floudas,et al.  Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption , 2017, J. Glob. Optim..

[62]  Ying Jiang,et al.  B-spline quasi-interpolation on sparse grids , 2011, J. Complex..

[63]  Dirk Pflüger,et al.  Hierarchical Gradient-Based Optimization with B-Splines on Sparse Grids , 2016 .

[64]  Antonello Monti,et al.  Dimension-Adaptive Sparse Grid Interpolation for Uncertainty Quantification in Modern Power Systems: Probabilistic Power Flow , 2016, IEEE Transactions on Power Systems.

[65]  Erich Novak,et al.  High dimensional polynomial interpolation on sparse grids , 2000, Adv. Comput. Math..

[66]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[67]  Christine A. Shoemaker,et al.  A quasi-multistart framework for global optimization of expensive functions using response surface models , 2013, J. Glob. Optim..

[68]  Selen Cremaschi,et al.  Adaptive sequential sampling for surrogate model generation with artificial neural networks , 2014, Comput. Chem. Eng..

[69]  Loo Hay Lee,et al.  Simulation optimization via kriging: a sequential search using expected improvement with computing budget constraints , 2013 .

[70]  Iftekhar A. Karimi,et al.  Smart Sampling Algorithm for Surrogate Model Development , 2017, Comput. Chem. Eng..

[71]  Luc Pronzato,et al.  Design of computer experiments: space filling and beyond , 2011, Statistics and Computing.

[72]  Charles Audet,et al.  A MADS Algorithm with a Progressive Barrier for Derivative-Free Nonlinear Programming , 2007 .

[73]  Marianthi G. Ierapetritou,et al.  A centroid-based sampling strategy for kriging global modeling and optimization , 2009 .

[74]  Nélida E. Echebest,et al.  A derivative-free method for solving box-constrained underdetermined nonlinear systems of equations , 2012, Appl. Math. Comput..