Model-Based Methods in Derivative-Free Nonsmooth Optimization

Derivative-free optimization (DFO) is the mathematical study of the optimization algorithms that do not use derivatives. One branch of DFO focuses on model-based DFO methods, where an approximation of the objective function is used to guide the optimization algorithm. Historically, model-based DFO has often assumed that the objective function is smooth, but unavailable analytically. However, recent progress has brought model-based DFO into the realm of nonsmooth optimization (NSO). In this chapter, we survey some of the progress of model-based DFO for nonsmooth functions. We begin with some historical context on model-based DFO. From there, we discuss methods for constructing models of smooth functions and their accuracy. This leads to modelling techniques for nonsmooth functions and a discussion on several frameworks for model-based DFO for NSO. We conclude the chapter with some of our opinions on profitable research directions in model-based DFO for NSO.

[1]  Robert Hooke,et al.  `` Direct Search'' Solution of Numerical and Statistical Problems , 1961, JACM.

[2]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[3]  D. Winfield,et al.  Function Minimization by Interpolation in a Data Table , 1973 .

[4]  Robert Mifflin,et al.  A superlinearly convergent algorithm for minimization without evaluating derivatives , 1975, Math. Program..

[5]  M. Powell.  A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation , 1994 .

[6]  Juan C. Meza,et al.  On the Use of Direct Search Methods for the Molecular Conformation Problem , 1994 .

[7]  M. L. MartinezScienti.  On the Use of Direct Search Methods for the Molecular Conformation Problem , 1994 .

[8]  T. Sauer,et al.  On multivariate Lagrange interpolation , 1995 .

[9]  P. Toint,et al.  An Algorithm using Quadratic Interpolation for Unconstrained Derivative Free Optimization , 1996 .

[10]  Margaret H. Wright,et al.  Direct search methods: Once scorned, now respectable , 1996 .

[11]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[12]  Virginia Torczon,et al.  On the Convergence of Pattern Search Algorithms , 1997, SIAM J. Optim..

[13]  C. Kelley,et al.  The Simplex Gradient and Noisy Optimization Problems , 1998 .

[14]  John E. Dennis,et al.  Optimization Using Surrogate Objectives on a Helicopter Test Example , 1998 .

[15]  Katya Scheinberg,et al.  A derivative free optimization algorithm in practice , 1998 .

[16]  John E. Dennis,et al.  Managing surrogate objectives to optimize a helicopter rotor design - Further experiments , 1998 .

[17]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[18]  Paul Tseng,et al.  Fortified-Descent Simplicial Search Method: A General Approach , 1999, SIAM J. Optim..

[19]  John E. Dennis,et al.  A framework for managing models in nonlinear optimization of computationally expensive functions , 1999 .

[20]  C. T. Kelley,et al.  Detection and Remediation of Stagnation in the Nelder--Mead Algorithm Using a Sufficient Decrease Condition , 1999, SIAM J. Optim..

[21]  I. D. Coope,et al.  A Convergent Variant of the Nelder–Mead Algorithm , 2002 .

[22]  M. J. D. Powell,et al.  UOBYQA: unconstrained optimization by quadratic approximation , 2002, Math. Program..

[23]  M. J. D. Powell,et al.  On trust region methods for unconstrained minimization without derivatives , 2003, Math. Program..

[24]  Tamara G. Kolda,et al.  Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods , 2003, SIAM Rev..

[25]  Lixing Han,et al.  On the convergence of the UOBYQA method , 2004 .

[26]  Fernando Nogueira,et al.  Pattern Search Methods for User-Provided Points: Application to Molecular Geometry Problems , 2004, SIAM J. Optim..

[27]  M. J. D. Powell,et al.  Least Frobenius norm updating of quadratic models that satisfy interpolation conditions , 2004, Math. Program..

[28]  R. Oeuvray,et al.  A New Derivative-Free Algorithm for the Medical Image Registration Problem , 2007 .

[29]  Christine A. Shoemaker,et al.  Comparison of function approximation, heuristic, and derivative‐based methods for automatic calibration of computationally expensive groundwater bioremediation models , 2005 .

[30]  Adrian S. Lewis,et al.  A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization , 2005, SIAM J. Optim..

[31]  Adrian S. Lewis,et al.  Estimating Tangent and Normal Cones Without Calculus , 2005, Math. Oper. Res..

[32]  R. Oeuvray.  Trust-region methods based on radial basis functions with application to biomedical imaging , 2005 .

[33]  Frank Vanden Berghen,et al.  CONDOR, a new parallel, constrained extension of Powell's UOBYQA algorithm: experimental results and comparison with the DFO algorithm , 2005 .

[34]  Robert Mifflin,et al.  A -algorithm for convex minimization , 2005, Math. Program..

[35]  Charles Audet,et al.  Mesh Adaptive Direct Search Algorithms for Constrained Optimization , 2006, SIAM J. Optim..

[36]  M. Powell.  The NEWUOA software for unconstrained optimization without derivatives , 2006 .

[37]  Luís N. Vicente,et al.  Using Sampling and Simplex Derivatives in Pattern Search Methods , 2007, SIAM J. Optim..

[38]  L. N. Vicente,et al.  Using simplex gradients of nonsmooth functions in direct search methods , 2008 .

[39]  Charles Audet,et al.  Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems , 2008 .

[40]  Donald J. Schuirmann,et al.  Sequential design approaches for bioequivalence studies with crossover designs , 2008, Pharmaceutical statistics.

[41]  Arnold Neumaier,et al.  SNOBFIT -- Stable Noisy Optimization by Branch and Fit , 2008, TOMS.

[42]  Katya Scheinberg,et al.  Geometry of interpolation sets in derivative free optimization , 2007, Math. Program..

[43]  Christine A. Shoemaker,et al.  ORBIT: Optimization by Radial Basis Function Interpolation in Trust-Regions , 2008, SIAM J. Sci. Comput..

[44]  A. Bagirov,et al.  Discrete Gradient Method: Derivative-Free Method for Nonsmooth Optimization , 2008 .

[45]  M. Powell.  The BOBYQA algorithm for bound constrained optimization without derivatives , 2009 .

[46]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[47]  Charles Audet,et al.  A MADS Algorithm with a Progressive Barrier for Derivative-Free Nonlinear Programming , 2007 .

[48]  Katya Scheinberg,et al.  Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical Points , 2009, SIAM J. Optim..

[49]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[50]  Humberto Rocha,et al.  Incorporating minimum Frobenius norm models in direct search , 2010, Comput. Optim. Appl..

[51]  Louis J. Durlofsky,et al.  Application of derivative-free methodologies to generally constrained oil production optimization problems , 2010, ICCS.

[52]  Charles Audet,et al.  Calculating Optimal Conditions for Alloy and Process Design Using Thermodynamic and Properties Databases, the FactSage Software and the Mesh Adaptive Direct Search (MADS) Algorithm , 2010 .

[53]  Krzysztof C. Kiwiel,et al.  A Nonderivative Version of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization , 2010, SIAM J. Optim..

[54]  Sébastien Le Digabel,et al.  Calculating all local minima on liquidus surfaces using the FactSage software and databases and the Mesh Adaptive Direct Search algorithm , 2011 .

[55]  Vincent Garnier,et al.  Snow Water Equivalent Estimation Using Blackbox Optimization , 2011 .

[56]  C. T. Kelley,et al.  Implicit Filtering , 2011 .

[57]  Christine A. Shoemaker,et al.  Global Convergence of Radial Basis Function Trust Region Derivative-Free Algorithms , 2011, SIAM J. Optim..

[58]  E. Renaud,et al.  Thermodynamic evaluation and optimization of the Li, Na, K, Mg, Ca, Sr // F, Cl reciprocal system , 2011 .

[59]  Warren Hare,et al.  A derivative-free approximate gradient sampling algorithm for finite minimax problems , 2013, Computational Optimization and Applications.

[60]  Warren Hare,et al.  Derivative-free optimization methods for finite minimax problems , 2013, Optim. Methods Softw..

[61]  Jeffrey Larson,et al.  Derivative-Free Optimization of Expensive Functions with Computational Error Using Weighted Regression , 2013, SIAM J. Optim..

[62]  Sébastien Le Digabel,et al.  Use of quadratic models with mesh-adaptive direct search for constrained black box optimization , 2011, Optim. Methods Softw..

[63]  Warren Hare,et al.  Exploiting Known Structures to Approximate Normal Cones , 2013, Math. Oper. Res..

[64]  Stefano Lucidi,et al.  A Linesearch-Based Derivative-Free Approach for Nonsmooth Constrained Optimization , 2014, SIAM J. Optim..

[65]  Charles Audet,et al.  Improving process representation in conceptual hydrological model calibration using climate simulations , 2014 .

[66]  Charles Audet,et al.  Reducing the Number of Function Evaluations in Mesh Adaptive Direct Search Algorithms , 2012, SIAM J. Optim..

[67]  Warren Hare,et al.  Derivative-Free Optimization Via Proximal Point Methods , 2014, J. Optim. Theory Appl..

[68]  Charles Audet,et al.  A Survey on Direct Search Methods for Blackbox Optimization and their Applications , 2014 .

[69]  Eduardo Camponogara,et al.  Derivative-free methods applied to daily production optimization of gas-lifted oil fields , 2015, Comput. Chem. Eng..

[70]  W. Hare,et al.  Optimizing damper connectors for adjacent buildings , 2015, 1511.02182.

[71]  Rommel G. Regis,et al.  The calculus of simplex gradients , 2015, Optim. Lett..

[72]  Árpád Bürmen,et al.  Mesh adaptive direct search with second directional derivative-based Hessian update , 2015, Comput. Optim. Appl..

[73]  Philippe L. Toint,et al.  A derivative-free trust-funnel method for equality-constrained nonlinear optimization , 2014, Computational Optimization and Applications.

[74]  Heinz H. Bauschke,et al.  A derivative-free comirror algorithm for convex optimization , 2015, Optim. Methods Softw..

[75]  W. Hare,et al.  Adaptive Interpolation Strategies in Derivative-Free Optimization: a case study , 2015, 1511.02794.

[76]  Stefan M. Wild,et al.  Manifold Sampling for ℓ1 Nonconvex Optimization , 2016, SIAM J. Optim..

[77]  Donald J. Schuirmann,et al.  Optimal adaptive sequential designs for crossover bioequivalence studies , 2015, Pharmaceutical statistics.

[78]  Charles Audet,et al.  Algorithmic Construction of the Subdifferential from Directional Derivatives , 2016 .

[79]  Charles Audet,et al.  Locally weighted regression models for surrogate-assisted design optimization , 2016 .

[80]  Stefan M. Wild,et al.  CONORBIT: constrained optimization by radial basis function interpolation in trust regions† , 2017, Optim. Methods Softw..

[81]  Charles Audet,et al.  Derivative-Free and Blackbox Optimization , 2017 .

[82]  W. Hare,et al.  Compositions of convex functions and fully linear models , 2016, Optim. Lett..

[83]  Warren Hare,et al.  Best practices for comparing optimization algorithms , 2017, Optimization and Engineering.

[84]  Juliane Müller,et al.  GOSAC: global optimization with surrogate approximation of constraints , 2017, J. Glob. Optim..

[85]  Katya Scheinberg,et al.  Methodologies and software for derivative-free optimization , 2017 .

[86]  Charles Audet,et al.  Efficient solution of quadratically constrained quadratic subproblems within the mesh adaptive direct search algorithm , 2017, Eur. J. Oper. Res..

[87]  Catherine Poissant.  Exploitation d’une structure monotone en recherche directepour l’optimisation de boîtes grises , 2018 .

[88]  Stefan M. Wild,et al.  Manifold Sampling for Optimization of Nonconvex Functions That Are Piecewise Linear Compositions of Smooth Components , 2018, SIAM J. Optim..

[89]  Charles Audet,et al.  A progressive barrier derivative-free trust-region algorithm for constrained optimization , 2016, Comput. Optim. Appl..

[90]  Rachael Tappenden,et al.  Efficient calculation of regular simplex gradients , 2017, Comput. Optim. Appl..

[91]  W. Hare,et al.  A derivative-free $\mathcal{VU}$-algorithm for convex finite-max problems , 2019, 1903.11184.

[92]  Michael Kokkolaras.  C. Audet and W. Hare: Derivative-free and blackbox optimization. Springer series in operations research and financial engineering , 2019 .

[93]  Warren Hare,et al.  A derivative-free 𝒱𝒰-algorithm for convex finite-max problems , 2019, Optim. Methods Softw..