Derivative-free optimization methods

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.

[1]  Saeed Ghadimi,et al.  Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming , 2013, SIAM J. Optim..

[2]  R. Durrett.  Probability: Theory and Examples , 1993 .

[3]  John L. Nazareth,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[4]  Charles Audet,et al.  Selection of variables in parallel space decomposition for the mesh adaptive , 2018 .

[5]  Sébastien Le Digabel,et al.  Algorithm xxx : NOMAD : Nonlinear Optimization with the MADS algorithm , 2010 .

[6]  Robert Michael Lewis,et al.  Pattern Search Algorithms for Bound Constrained Minimization , 1999, SIAM J. Optim..

[7]  Frank Vanden Berghen,et al.  CONDOR, a new parallel, constrained extension of Powell's UOBYQA algorithm: experimental results and comparison with the DFO algorithm , 2005 .

[8]  N. V. Banichuk,et al.  The solution of variational and boundary value problems by the method of local variations , 1966 .

[9]  Florian Jarre,et al.  Calibration by optimization without using derivatives , 2016 .

[10]  Peter Richt'arik,et al.  Simple Complexity Analysis of Simplified Direct Search , 2014, 1410.0390.

[11]  Adam Tauman Kalai,et al.  Online convex optimization in the bandit setting: gradient descent without a gradient , 2004, SODA '05.

[12]  Ruobing Chen,et al.  Stochastic derivative-free optimization of noisy functions , 2015 .

[13]  Peter W. Glynn,et al.  On Sampling Rates in Simulation-Based Recursions , 2018, SIAM J. Optim..

[14]  M. J. D. Powell.  On the convergence of a wide range of trust region methods for unconstrained optimization , 2010 .

[15]  H. L. Anderson.  Scientific uses of the MANIAC , 1986 .

[16]  Robert Mifflin,et al.  A superlinearly convergent algorithm for minimization without evaluating derivatives , 1975, Math. Program..

[17]  Philippe L. Toint,et al.  Numerical experience with a derivative-free trust-funnel method for nonlinear optimization problems with general nonlinear constraints , 2016, Optim. Methods Softw..

[18]  Andrew D. Back,et al.  Radial Basis Functions , 2001 .

[19]  Benedetta Morini,et al.  Approximate norm descent methods for constrained nonlinear systems , 2018, Math. Comput..

[20]  Krzysztof C. Kiwiel,et al.  A Nonderivative Version of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization , 2010, SIAM J. Optim..

[21]  Maria Grazia Gasparo,et al.  Generalized Pattern Search methods for a class of nonsmooth optimization problems with structure , 2009 .

[22]  Giacomo Nannicini,et al.  MILP models for the selection of a small set of well-distributed points , 2017, Oper. Res. Lett..

[23]  I. D. Coope,et al.  A Convergent Variant of the Nelder–Mead Algorithm , 2002 .

[24]  Yin Tat Lee,et al.  Kernel-based methods for bandit convex optimization , 2016, STOC.

[25]  J. Kiefer,et al.  Stochastic Estimation of the Maximum of a Regression Function , 1952 .

[26]  Raghu Pasupathy,et al.  ASTRO-DF: A Class of Adaptive Sampling Trust-Region Algorithms for Derivative-Free Stochastic Optimization , 2016, SIAM J. Optim..

[27]  Stefan M. Wild,et al.  Estimating Derivatives of Noisy Simulations , 2012, TOMS.

[28]  P. D. Conejo,et al.  Global convergence of trust-region algorithms for convex constrained minimization without derivatives , 2013, Appl. Math. Comput..

[29]  Masha Sosonkina,et al.  Design and implementation of a massively parallel version of DIRECT , 2008, Comput. Optim. Appl..

[30]  Anke Tröltzsch,et al.  A sequential quadratic programming algorithm for equality-constrained optimization without derivatives , 2014, Optimization Letters.

[31]  L. Grippo,et al.  Global convergence and stabilization of unconstrained minimization methods without derivatives , 1988 .

[32]  Nélida E. Echebest,et al.  Active-set strategy in Powell's method for optimization without derivatives , 2011 .

[33]  Vishnu Narayanan,et al.  A scaled conjugate gradient based direct search algorithm for high dimensional box constrained derivative free optimization , 2019, 1901.05215.

[34]  Warren Hare,et al.  A derivative-free 𝒱𝒰-algorithm for convex finite-max problems , 2019, Optim. Methods Softw..

[35]  C. T. Kelley,et al.  An Implicit Filtering Algorithm for Optimization of Functions with Many Local Minima , 1995, SIAM J. Optim..

[36]  Nicholas I. M. Gould,et al.  On the Oracle Complexity of First-Order and Derivative-Free Algorithms for Smooth Nonconvex Minimization , 2012, SIAM J. Optim..

[37]  E. W. Karas,et al.  Global convergence of a derivative-free inexact restoration filter algorithm for nonlinear programming , 2017 .

[38]  J. Blum.  Multidimensional Stochastic Approximation Methods , 1954 .

[39]  J. Sacks.  Asymptotic Distribution of Stochastic Approximation Procedures , 1958 .

[40]  Victor Picheny,et al.  Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian , 2016, NIPS.

[41]  Thomas J. Asaki,et al.  Using QR Decomposition to Obtain a New Instance of Mesh Adaptive Direct Search with Uniformly Distributed Polling Directions , 2013, J. Optim. Theory Appl..

[42]  Nikolaos V. Sahinidis,et al.  Derivative-free optimization: a review of algorithms and comparison of software implementations , 2013, J. Glob. Optim..

[43]  M. Urner.  Scattered Data Approximation , 2016 .

[44]  Michael Wetter,et al.  Precision Control for Generalized Pattern Search Algorithms with Adaptive Precision Function Evaluations , 2003, SIAM J. Optim..

[45]  Stefan M. Wild,et al.  Manifold Sampling for Optimization of Nonconvex Functions That Are Piecewise Linear Compositions of Smooth Components , 2018, SIAM J. Optim..

[46]  M. Powell.  On updating the inverse of a KKT matrix 1 , 2004 .

[47]  Jia Yuan Yu,et al.  Lipschitz Bandits without the Lipschitz Constant , 2011, ALT.

[48]  Rommel G. Regis,et al.  On the properties of positive spanning sets and positive bases , 2016 .

[49]  M. J. D. Powell,et al.  Beyond symmetric Broyden for updating quadratic models in minimization without derivatives , 2013, Math. Program..

[50]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[51]  M. J. Box.  A New Method of Constrained Optimization and a Comparison With Other Methods , 1965, Comput. J..

[52]  J. Dennis,et al.  Filter Pattern Search Algorithms for Mixed Variable Constrained Optimization Problems , 2004 .

[53]  Ambuj Tewari,et al.  On the Universality of Online Mirror Descent , 2011, NIPS.

[54]  Sven Leyffer,et al.  A method for convex black-box integer global optimization , 2019, J. Glob. Optim..

[55]  Jean Bigeon,et al.  Performance indicators in multiobjective optimization , 2018, Eur. J. Oper. Res..

[56]  Jong-hyun Ryu,et al.  A Trust-Region Algorithm for Bi-Objective Stochastic Optimization , 2011, ICCS.

[57]  Robert Michael Lewis,et al.  A Direct Search Approach to Nonlinear Programming Problems Using an Augmented Lagrangian Method with Explicit Treatment of Linear Constraints , 2010 .

[58]  Arnold Neumaier,et al.  VXQR: derivative-free unconstrained optimization based on QR factorizations , 2011, Soft Comput..

[59]  Juliane Müller,et al.  Surrogate Optimization of Computationally Expensive Black-Box Problems with Hidden Constraints , 2019, INFORMS J. Comput..

[60]  C. T. Kelley,et al.  Superlinear Convergence and Implicit Filtering , 1999, SIAM J. Optim..

[61]  Jeffrey Larson,et al.  Derivative-Free Optimization of Expensive Functions with Computational Error Using Weighted Regression , 2013, SIAM J. Optim..

[62]  Tamara G. Kolda,et al.  Algorithm 856: APPSPACK 4.0: asynchronous parallel pattern search for derivative-free optimization , 2006, TOMS.

[63]  M. J. D. Powell,et al.  On the convergence of trust region algorithms for unconstrained minimization without derivatives , 2012, Comput. Optim. Appl..

[64]  Karen Willcox,et al.  A Provably Convergent Multifidelity Optimization Algorithm not Requiring High-Fidelity Derivatives , 2010 .

[65]  Arnold Neumaier,et al.  SNOBFIT -- Stable Noisy Optimization by Branch and Fit , 2008, TOMS.

[66]  Vianney Perchet,et al.  Highly-Smooth Zero-th Order Online Optimization , 2016, COLT.

[67]  Zaikun Zhang,et al.  Sobolev seminorm of quadratic functions with applications to derivative-free optimization , 2011, Math. Program..

[68]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[69]  M. Powell.  The BOBYQA algorithm for bound constrained optimization without derivatives , 2009 .

[70]  Todd D. Plantenga,et al.  HOPSPACK 2.0 user manual. , 2009 .

[71]  Huifen Chen,et al.  Stochastic root finding via retrospective approximation , 2001 .

[72]  Serge Gratton,et al.  A Merit Function Approach for Direct Search , 2014, SIAM J. Optim..

[73]  Bin Gu,et al.  Zeroth-order Asynchronous Doubly Stochastic Algorithm with Variance Reduction , 2016, ArXiv.

[74]  Stefan M. Wild,et al.  Derivative-free robust optimization by outer approximations , 2018, Math. Program..

[75]  A. Neumaier.  Complete search in continuous global optimization and constraint satisfaction , 2004, Acta Numerica.

[76]  Luís N. Vicente,et al.  Using Sampling and Simplex Derivatives in Pattern Search Methods , 2007, SIAM J. Optim..

[77]  M. Powell.  The use of Band Matrices for Second Derivative Approximations in Trust Region Algorithms , 1998 .

[78]  Marco Sciandrone,et al.  Sequential Penalty Derivative-Free Methods for Nonlinear Constrained Optimization , 2010, SIAM J. Optim..

[79]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[80]  Rommel G. Regis,et al.  The calculus of simplex gradients , 2015, Optim. Lett..

[81]  Luigi Grippo,et al.  Nonmonotone derivative-free methods for nonlinear equations , 2007, Comput. Optim. Appl..

[82]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[83]  A. A. Zhigli︠a︡vskiĭ,et al.  Theory of Global Random Search , 1991 .

[84]  Rémi Munos,et al.  Optimistic Optimization of Deterministic Functions , 2011, NIPS 2011.

[85]  Raghu Pasupathy,et al.  On Choosing Parameters in Retrospective-Approximation Algorithms for Stochastic Root Finding and Simulation Optimization , 2010, Oper. Res..

[86]  Jorge Nocedal,et al.  On the geometry phase in model-based algorithms for derivative-free optimization , 2009, Optim. Methods Softw..

[87]  Paul Tseng,et al.  Fortified-Descent Simplicial Search Method: A General Approach , 1999, SIAM J. Optim..

[88]  Katya Scheinberg,et al.  Recent progress in unconstrained nonlinear optimization without derivatives , 1997, Math. Program..

[89]  M. T. Wasan.  Stochastic Approximation , 1969 .

[90]  Philip E. Gill,et al.  Practical optimization , 1981 .

[91]  Robert Michael Lewis,et al.  On the Local Convergence of Pattern Search , 2003, SIAM J. Optim..

[92]  Trond Steihaug,et al.  A New Generating Set Search Algorithm for Partially Separable Functions , 2010 .

[93]  Charles Audet,et al.  A Survey on Direct Search Methods for Blackbox Optimization and their Applications , 2014 .

[94]  Loïc Le Gratiet,et al.  Cokriging-Based Sequential Design Strategies Using Fast Cross-Validation Techniques for Multi-Fidelity Computer Codes , 2015, Technometrics.

[95]  Trond Steihaug,et al.  A generating set search method using curvature information , 2007, Comput. Optim. Appl..

[96]  R. Agrawal.  Sample mean based index policies by O(log n) regret for the multi-armed bandit problem , 1995, Advances in Applied Probability.

[97]  Charles Audet,et al.  Linear equalities in blackbox optimization , 2014, Comput. Optim. Appl..

[98]  Tito Homem-de-Mello,et al.  Monte Carlo sampling-based methods for stochastic optimization , 2014 .

[99]  Ludmil T. Zikatanov,et al.  Algebraic multigrid methods * , 2016, Acta Numerica.

[100]  Andrew R. Conn,et al.  On the local convergence of a derivative-free algorithm for least-squares minimization , 2012, Comput. Optim. Appl..

[101]  Dimitris Bertsimas,et al.  Robust Optimization for Unconstrained Simulation-Based Problems , 2010, Oper. Res..

[102]  Owen J. Eslinger,et al.  Algorithms for Noisy Problems in Gas Transmission Pipeline Optimization , 2001 .

[103]  Alexander V. Gasnikov,et al.  Stochastic online optimization. Single-point and multi-point non-linear multi-armed bandits. Convex and strongly-convex case , 2017, Autom. Remote. Control..

[104]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[105]  Gang George Yin,et al.  Budget-Dependent Convergence Rate of Stochastic Approximation , 1995, SIAM J. Optim..

[106]  Stefan M. Wild,et al.  CONORBIT: constrained optimization by radial basis function interpolation in trust regions† , 2017, Optim. Methods Softw..

[107]  D. Sculley,et al.  Google Vizier: A Service for Black-Box Optimization , 2017, KDD.

[108]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[109]  Christine A. Shoemaker,et al.  Global Convergence of Radial Basis Function Trust-Region Algorithms for Derivative-Free Optimization , 2013, SIAM Rev..

[110]  Stefan M. Wild,et al.  Asynchronously parallel optimization solver for finding multiple minima , 2018, Math. Program. Comput..

[111]  Sébastien Le Digabel,et al.  Use of quadratic models with mesh-adaptive direct search for constrained black box optimization , 2011, Optim. Methods Softw..

[112]  Robert Michael Lewis,et al.  Pattern Search Methods for Linearly Constrained Minimization , 1999, SIAM J. Optim..

[113]  F. Downton.  Stochastic Approximation , 1969, Nature.

[114]  Christine A. Shoemaker,et al.  SO-I: a surrogate model algorithm for expensive nonlinear integer programming problems including global optimization applications , 2014, J. Glob. Optim..

[115]  Charles Audet,et al.  Mesh Adaptive Direct Search Algorithms for Constrained Optimization , 2006, SIAM J. Optim..

[116]  M. Powell.  Developments of NEWUOA for minimization without derivatives , 2008 .

[117]  Nicholas I. M. Gould,et al.  Nonlinear programming without a penalty function or a filter , 2010, Math. Program..

[118]  Andreas Griewank,et al.  On Lipschitz optimization based on gray-box piecewise linearization , 2016, Math. Program..

[119]  Douglass J. Wilde,et al.  Optimal Condenser Design by Geometric Programming , 1967 .

[120]  Tamara G. Kolda,et al.  Stationarity Results for Generating Set Search for Linearly Constrained Optimization , 2006, SIAM J. Optim..

[121]  J. Mockus.  Bayesian Approach to Global Optimization: Theory and Applications , 1989 .

[122]  José Mario Martínez,et al.  Constrained derivative-free optimization on thin domains , 2012, Journal of Global Optimization.

[123]  Lixing Han,et al.  On the convergence of the UOBYQA method , 2004 .

[124]  Rong Jin,et al.  Online Bandit Learning for a Special Class of Non-Convex Losses , 2015, AAAI.

[125]  J. Spall,et al.  Simulation-Based Optimization with Stochastic Approximation Using Common Random Numbers , 1999 .

[126]  José Mario Martínez,et al.  Spectral residual method without gradient information for solving large-scale nonlinear systems of equations , 2006, Math. Comput..

[127]  Charles Audet,et al.  Reducing the Number of Function Evaluations in Mesh Adaptive Direct Search Algorithms , 2012, SIAM J. Optim..

[128]  Humberto Rocha,et al.  Incorporating minimum Frobenius norm models in direct search , 2010, Comput. Optim. Appl..

[129]  G. R. Hext,et al.  Sequential Application of Simplex Designs in Optimisation and Evolutionary Operation , 1962 .

[130]  Christine A. Shoemaker,et al.  Derivative-free optimization algorithms for computationally expensive functions , 2009 .

[131]  Irina S. Dolinskaya,et al.  A Derivative-Free Trust-Region Algorithm for the Optimization of Functions Smoothed via Gaussian Convolution Using Adaptive Multiple Importance Sampling , 2018, SIAM J. Optim..

[132]  Guannan Zhang,et al.  Stochastic finite element methods for partial differential equations with random input data* , 2014, Acta Numerica.

[133]  Katya Scheinberg,et al.  Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales , 2016, INFORMS Journal on Optimization.

[134]  Andrew R. Conn,et al.  Bilevel derivative-free optimization and its application to robust optimization , 2012, Optim. Methods Softw..

[135]  Charles Audet,et al.  Mesh adaptive direct search algorithms for mixed variable optimization , 2007, Optim. Lett..

[136]  Ubaldo M. García-Palomares,et al.  New Sequential and Parallel Derivative-Free Algorithms for Unconstrained Minimization , 2002, SIAM J. Optim..

[137]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[138]  C. Kelley,et al.  The Simplex Gradient and Noisy Optimization Problems , 1998 .

[139]  Juan C. Meza,et al.  A Class of Trust-Region Methods for Parallel Optimization , 1999, SIAM J. Optim..

[140]  P. Toint,et al.  A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds , 1991 .

[141]  Ambuj Tewari,et al.  Improved Regret Guarantees for Online Smooth Convex Optimization with Bandit Feedback , 2011, AISTATS.

[142]  Heinz H. Bauschke,et al.  A derivative-free comirror algorithm for convex optimization , 2015, Optim. Methods Softw..

[143]  Xiaojun Chen,et al.  A Smoothing Direct Search Method for Monte Carlo-Based Bound Constrained Composite Nonsmooth Optimization , 2018, SIAM J. Sci. Comput..

[144]  Rik Sferra,et al.  Oral history , 1998, SIGGRAPH '98.

[145]  Coralia Cartis,et al.  Improving the Flexibility and Robustness of Model-based Derivative-free Optimization Solvers , 2018, ACM Trans. Math. Softw..

[146]  H. H. Rosenbrock,et al.  An Automatic Method for Finding the Greatest or Least Value of a Function , 1960, Comput. J..

[147]  Hans-Martin Gutmann,et al.  A Radial Basis Function Method for Global Optimization , 2001, J. Glob. Optim..

[148]  L. Vicente,et al.  SID-PSM: A PATTERN SEARCH METHOD GUIDED BY SIMPLEX DERIVATIVES FOR USE IN DERIVATIVE-FREE OPTIMIZATION , 2010 .

[149]  Hong Wan,et al.  Stochastic Trust-Region Response-Surface Method (STRONG) - A New Response-Surface Framework for Simulation Optimization , 2013, INFORMS J. Comput..

[150]  Rachael Tappenden,et al.  Efficient calculation of regular simplex gradients , 2017, Comput. Optim. Appl..

[151]  G. Peckham,et al.  A New Method for Minimising a Sum of Squares Without Calculating Gradients , 1970, Comput. J..

[152]  Charles Audet,et al.  A mesh adaptive direct search algorithm for multiobjective optimization , 2009, Eur. J. Oper. Res..

[153]  Jeffrey C. Lagarias,et al.  Convergence of the Restricted Nelder-Mead Algorithm in Two Dimensions , 2011, SIAM J. Optim..

[154]  Sujin Kim,et al.  Convergence properties of direct search methods for stochastic optimization , 2010, Proceedings of the 2010 Winter Simulation Conference.

[155]  Katya Scheinberg,et al.  Convergence of Trust-Region Methods Based on Probabilistic Models , 2013, SIAM J. Optim..

[156]  S. Nash.  A multigrid approach to discretized optimization problems , 2000 .

[157]  Luís N. Vicente,et al.  Direct Multisearch for Multiobjective Optimization , 2011, SIAM J. Optim..

[158]  Peter Auer,et al.  Using Confidence Bounds for Exploitation-Exploration Trade-offs , 2003, J. Mach. Learn. Res..

[159]  Herbert K. H. Lee,et al.  Bayesian Guided Pattern Search for Robust Local Optimization , 2009, Technometrics.

[160]  Y. Marzouk,et al.  A trust-region method for derivative-free nonlinear constrained stochastic optimization , 2017, 1703.04156.

[161]  Matthew Joseph Menickelly,et al.  Random Models in Nonlinear Optimization , 2017 .

[162]  Katya Scheinberg,et al.  On the convergence of derivative-free methods for unconstrained optimization , 1997 .

[163]  Clóvis C. Gonzaga,et al.  A Globally Convergent Filter Method for Nonlinear Programming , 2003, SIAM J. Optim..

[164]  Robert Hooke,et al.  `` Direct Search'' Solution of Numerical and Statistical Problems , 1961, JACM.

[165]  Stefano Lucidi,et al.  A Derivative-Free Approach to Constrained Multiobjective Nonsmooth Optimization , 2016, SIAM J. Optim..

[166]  Lois Curfman McInnes,et al.  TAO 3.10 Users Manual , 2018 .

[167]  Ya-Xiang Yuan,et al.  A derivative-free trust-region algorithm for composite nonsmooth optimization , 2014, Computational and Applied Mathematics.

[168]  Andy J. Keane,et al.  Engineering Design via Surrogate Modelling - A Practical Guide , 2008 .

[169]  Virginia Torczon,et al.  On the Convergence of the Multidirectional Search Algorithm , 1991, SIAM J. Optim..

[170]  D. Finkel,et al.  Convergence analysis of the direct algorithm , 2004 .

[171]  Andrea Walther,et al.  Recent Advances in Algorithmic Differentiation , 2012 .

[172]  Liyi Dai,et al.  Convergence rates of finite difference stochastic approximation algorithms part II: implementation via common random numbers , 2016, Commercial + Scientific Sensing and Imaging.

[173]  R. Oeuvray.  Trust-region methods based on radial basis functions with application to biomedical imaging , 2005 .

[174]  M. J. D. Powell,et al.  A Method for Minimizing a Sum of Squares of Non-Linear Functions Without Calculating Derivatives , 1965, Comput. J..

[175]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[176]  Benjamin Naumann,et al.  Computational Methods In Optimization A Unified Approach , 2016 .

[177]  Marco Sciandrone,et al.  A Derivative-Free Algorithm for Linearly Constrained Finite Minimax Problems , 2006, SIAM J. Optim..

[178]  Michael L. Overton,et al.  Gradient Sampling Methods for Nonsmooth Optimization , 2018, Numerical Nonsmooth Optimization.

[179]  C. T. Kelley,et al.  Implicit Filtering and Nonlinear Least Squares Problems , 2001, System Modelling and Optimization.

[180]  M. Montaz Ali,et al.  A trust-region-based derivative free algorithm for mixed integer programming , 2014, Computational Optimization and Applications.

[181]  Luís N. Vicente,et al.  Worst case complexity of direct search under convexity , 2016, Math. Program..

[182]  Philippe L. Toint,et al.  Optimizing partially separable functions without derivatives , 2005, Optim. Methods Softw..

[183]  Junmei Ma,et al.  Pattern search methods for finite minimax problems , 2010 .

[184]  G. T. Timmer,et al.  Stochastic global optimization methods part I: Clustering methods , 1987, Math. Program..

[185]  Marco Sciandrone,et al.  A Derivative-Free Algorithm for Bound Constrained Optimization , 2002, Comput. Optim. Appl..

[186]  S. Lucidi,et al.  An algorithmic framework based on primitive directions and nonmonotone line searches for black box problems with integer variables , 2018 .

[187]  Stefan M. Wild,et al.  Manifold Sampling for ℓ1 Nonconvex Optimization , 2016, SIAM J. Optim..

[188]  Serge Gratton,et al.  Direct Search Based on Probabilistic Descent , 2015, SIAM J. Optim..

[189]  Árpád Bürmen,et al.  Mesh adaptive direct search with second directional derivative-based Hessian update , 2015, Comput. Optim. Appl..

[190]  Michael W. Trosset,et al.  Quasi-newton methods for stochastic optimization and proximity-based methods for disparate information fusion , 2012 .

[191]  Shih-Ping Han.  A globally convergent method for nonlinear programming , 1975 .

[192]  M. Fukushima,et al.  A derivative-free line search and global convergence of Broyden-like method for nonlinear equations , 2000 .

[193]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[194]  Juliane Müller.  MISO: mixed-integer surrogate optimization framework , 2016 .

[195]  D. Winfield,et al.  Function Minimization by Interpolation in a Data Table , 1973 .

[196]  Krishnakumar Balasubramanian,et al.  Zeroth-Order Nonconvex Stochastic Optimization: Handling Constraints, High Dimensionality, and Saddle Points , 2018, Foundations of Computational Mathematics.

[197]  Taimoor Akhtar,et al.  Efficient Hyperparameter Optimization for Deep Learning Algorithms Using Deterministic RBF Surrogates , 2016, AAAI.

[198]  Saeed Ghadimi,et al.  Conditional gradient type methods for composite nonlinear and stochastic optimization , 2016, Math. Program..

[199]  Katya Scheinberg,et al.  Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical Points , 2009, SIAM J. Optim..

[200]  L. N. Vicente,et al.  Geometry of sample sets in derivative-free optimization: polynomial regression and underdetermined interpolation , 2008 .

[201]  Y. Marzouk,et al.  NOWPAC: A provably convergent nonlinear optimizer with path-augmented constraints for noisy regimes , 2014 .

[202]  Charles Audet,et al.  Generalized pattern searches with derivative information , 2002, Math. Program..

[203]  J. Dennis,et al.  MANAGING APPROXIMATION MODELS IN OPTIMIZATION , 2007 .

[204]  L. N. Vicente,et al.  Using simplex gradients of nonsmooth functions in direct search methods , 2008 .

[205]  L. Vicente,et al.  A second-order globally convergent direct-search method and its worst-case complexity , 2016 .

[206]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[207]  R. Jennrich,et al.  Dud, A Derivative-Free Algorithm for Nonlinear Least Squares , 1978 .

[208]  G. T. Timmer,et al.  Stochastic global optimization methods part II: Multi level methods , 1987, Math. Program..

[209]  Carl Tim Kelley,et al.  Iterative methods for optimization , 1999, Frontiers in applied mathematics.

[210]  Michael C. Ferris,et al.  Variable-Number Sample-Path Optimization , 2008, Math. Program..

[211]  C. T. Kelley,et al.  Implicit Filtering , 2011 .

[212]  A Stochastic Line Search Method with Convergence Rate Analysis , 2018, 1807.07994.

[213]  Trond Steihaug,et al.  A subclass of generating set search with convergence to second-order stationary points , 2014, Optim. Methods Softw..

[214]  M. J. D. Powell,et al.  UOBYQA: unconstrained optimization by quadratic approximation , 2002, Math. Program..

[215]  Ya-Xiang Yuan,et al.  Conditions for convergence of trust region algorithms for nonsmooth optimization , 1985, Math. Program..

[216]  Robert L. Smith,et al.  Pure adaptive search in global optimization , 1992, Math. Program..

[217]  Eduard A. Gorbunov,et al.  An Accelerated Method for Derivative-Free Smooth Stochastic Convex Optimization , 2018, SIAM J. Optim..

[218]  Lixing Han,et al.  Implementing the Nelder-Mead simplex algorithm with adaptive parameters , 2010, Computational Optimization and Applications.

[219]  L. Cooper,et al.  Sequential Search: A Method for Solving Constrained Optimization Problems , 1965, JACM.

[220]  R. Pasupathy,et al.  A Guide to Sample Average Approximation , 2015 .

[221]  Charles Audet,et al.  A method for stochastic constrained optimization using derivative-free surrogate pattern search and collocation , 2010, J. Comput. Phys..

[222]  R. Fletcher,et al.  Function Minimization Without Evaluating Derivatives - a Review , 1965, Comput. J..

[223]  Warren Hare,et al.  A derivative-free approximate gradient sampling algorithm for finite minimax problems , 2013, Comput. Optim. Appl..

[224]  Jorge Nocedal,et al.  Wedge trust region methods for derivative free optimization , 2002, Math. Program..

[225]  M. J. D. Powell,et al.  A View of Unconstrained Minimization Algorithms that Do Not Require Derivatives , 1975, TOMS.

[226]  Liyi Dai,et al.  Convergence rates of finite difference stochastic approximation algorithms part I: general sampling , 2015, Commercial + Scientific Sensing and Imaging.

[227]  Katya Scheinberg,et al.  Global convergence rate analysis of unconstrained optimization methods based on probabilistic models , 2015, Mathematical Programming.

[228]  Daniel E. Finkel,et al.  Additive Scaling and the DIRECT Algorithm , 2006, J. Glob. Optim..

[229]  Per-Magnus Olsson,et al.  Methods for Network Optimization and Parallel Derivative-free Optimization , 2014 .

[230]  Csaba Szepesvári,et al.  –armed Bandits , 2022 .

[231]  Nicholas I. M. Gould,et al.  Sharp worst-case evaluation complexity bounds for arbitrary-order nonconvex optimization with inexpensive constraints , 2018, SIAM J. Optim..

[232]  W. Hare,et al.  Compositions of convex functions and fully linear models , 2016, Optim. Lett..

[233]  Benjamin Van Roy,et al.  An Information-Theoretic Analysis of Thompson Sampling , 2014, J. Mach. Learn. Res..

[234]  C. T. Kelley,et al.  Detection and Remediation of Stagnation in the Nelder--Mead Algorithm Using a Sufficient Decrease Condition , 1999, SIAM J. Optim..

[235]  Warren Hare,et al.  Exploiting Known Structures to Approximate Normal Cones , 2013, Math. Oper. Res..

[236]  Bernard Ghanem,et al.  A Stochastic Derivative-Free Optimization Method with Importance Sampling: Theory and Learning to Control , 2019, AAAI.

[237]  Susan R. Hunter,et al.  ASTRO-DF: Adaptive sampling trust-region optimization algorithms, heuristics, and numerical experience , 2016, 2016 Winter Simulation Conference (WSC).

[238]  A. Bagirov,et al.  Discrete Gradient Method: Derivative-Free Method for Nonsmooth Optimization , 2008 .

[239]  Dimitris Bertsimas,et al.  Robust optimization with simulated annealing , 2010, J. Glob. Optim..

[240]  Gang Yang,et al.  MrDIRECT: a multilevel robust DIRECT algorithm for global optimization problems , 2015, J. Glob. Optim..

[241]  John E. Dennis,et al.  Direct Search Methods on Parallel Machines , 1991, SIAM J. Optim..

[242]  A. L. Custódio,et al.  MultiGLODS: global and local multiobjective optimization using direct search , 2018, Journal of Global Optimization.

[243]  J. Blum.  Approximation Methods which Converge with Probability one , 1954 .

[244]  Charles Audet,et al.  Multiobjective Optimization Through a Series of Single-Objective Formulations , 2008, SIAM J. Optim..

[245]  Luís Nunes Vicente,et al.  Trust-Region Methods Without Using Derivatives: Worst Case Complexity and the NonSmooth Case , 2016, SIAM J. Optim..

[246]  Pierre Hansen,et al.  On the Number of Iterations of Piyavskii's Global Optimization Algorithm , 1991, Math. Oper. Res..

[247]  Stefan M. Wild.  Chapter 40: POUNDERS in TAO: Solving Derivative-Free Nonlinear Least-Squares Problems with POUNDERS , 2017 .

[248]  E. Polak,et al.  Computational methods in optimization : a unified approach , 1972 .

[249]  Janez Puhan,et al.  Grid Restrained Nelder-Mead Algorithm , 2006, Comput. Optim. Appl..

[250]  Ohad Shamir,et al.  An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback , 2015, J. Mach. Learn. Res..

[251]  Serge Gratton,et al.  Direct search based on probabilistic feasible descent for bound and linearly constrained problems , 2019, Computational Optimization and Applications.

[252]  Tony Pourmohamad,et al.  Combining Multivariate Stochastic Process Models with Filter Methods for Constrained Optimization , 2016 .

[253]  J. Dennis,et al.  Derivative free analogues of the Levenberg-Marquardt and Gauss algorithms for nonlinear least squares approximation , 1971 .

[254]  Shalabh Bhatnagar,et al.  Kiefer-Wolfowitz Algorithm , 2013 .

[255]  Pedro S. Rodríguez-Hernández,et al.  On sequential and parallel non-monotone derivative-free algorithms for box constrained optimization , 2013, Optim. Methods Softw..

[256]  Stefano Lucidi,et al.  A Linesearch-Based Derivative-Free Approach for Nonsmooth Constrained Optimization , 2014, SIAM J. Optim..

[257]  Krishnakumar Balasubramanian,et al.  Zeroth-order (Non)-Convex Stochastic Optimization via Conditional Gradient and Gradient Updates , 2018, NeurIPS.

[258]  Gui-Hua Lin,et al.  Bilevel direct search method for leader-follower problems and application in health insurance , 2014, Comput. Oper. Res..

[259]  Daniel J. Woods,et al.  Optimization on Microcomputers: The Nelder-Mead Simplex Algorithm , 1985 .

[260]  Katya Scheinberg,et al.  Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization , 2012, Mathematical Programming.

[261]  M. L. Schuverdt,et al.  An inexact restoration derivative-free filter method for nonlinear programming , 2015, Computational and Applied Mathematics.

[262]  J. M. Mart.  Constrained Derivative-Free Optimization on Thin Domains , 2011 .

[263]  L. Gerencsér.  Rate of convergence of moments of Spall's SPSA method , 1997, 1997 European Control Conference (ECC).

[264]  Arnold Neumaier,et al.  Global Optimization by Multilevel Coordinate Search , 1999, J. Glob. Optim..

[265]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[266]  Matthias Ehrgott,et al.  Multicriteria Optimization , 2005 .

[267]  M. Powell.  On the Lagrange functions of quadratic models that are defined by interpolation* , 2001 .

[268]  Nikolaos V. Sahinidis,et al.  Simulation optimization: a review of algorithms and applications , 2014, 4OR.

[269]  Herbert Hamers,et al.  Constrained optimization involving expensive function evaluations: A sequential approach , 2005, Eur. J. Oper. Res..

[270]  Stefan M. Wild,et al.  A batch, derivative-free algorithm for finding multiple local minima , 2016 .

[271]  Stefano Lucidi,et al.  Derivative-free methods for bound constrained mixed-integer optimization , 2011, Computational Optimization and Applications.

[272]  Sébastien Le Digabel,et al.  The mesh adaptive direct search algorithm with treed Gaussian process surrogates , 2011 .

[273]  Elizabeth W. Karas,et al.  A trust-region derivative-free algorithm for constrained optimization , 2015, Optim. Methods Softw..

[274]  J. S. Ivey,et al.  Nelder-Mead simplex modifications for simulation optimization , 1996 .

[275]  Prasanna Balaprakash,et al.  DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks , 2018, 2018 IEEE 25th International Conference on High Performance Computing (HiPC).

[276]  Katya Scheinberg,et al.  On the local convergence of a derivative-free algorithm for least-squares minimization , 2010, Computational Optimization and Applications.

[277]  M. Powell.  The NEWUOA software for unconstrained optimization without derivatives , 2006 .

[278]  Charles Audet.  Convergence Results for Generalized Pattern Search Algorithms are Tight , 2004 .

[279]  Charles Audet,et al.  Convergence of Mesh Adaptive Direct Search to Second-Order Stationary Points , 2006, SIAM J. Optim..

[280]  Emanuele Frandi,et al.  Coordinate search algorithms in multilevel optimization , 2014, Optim. Methods Softw..

[281]  José Mario Martínez,et al.  Inexact Restoration Method for Derivative-Free Optimization with Smooth Constraints , 2013, SIAM J. Optim..

[282]  Sham M. Kakade,et al.  Stochastic Convex Optimization with Bandit Feedback , 2011, SIAM J. Optim..

[283]  Masha Sosonkina,et al.  Remark on Algorithm 897 , 2015 .

[284]  R. Regis.  Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points , 2014 .

[285]  Xiaojun Chen,et al.  Optimization with hidden constraints and embedded Monte Carlo computations , 2016 .

[286]  Elad Hazan,et al.  Logistic Regression: Tight Bounds for Stochastic and Online Optimization , 2014, COLT.

[287]  N. M. Alexandrov,et al.  A trust-region framework for managing the use of approximation models in optimization , 1997 .

[288]  Christine A. Shoemaker,et al.  A Stochastic Radial Basis Function Method for the Global Optimization of Expensive Functions , 2007, INFORMS J. Comput..

[289]  Stefan M. Wild,et al.  Do you trust derivatives or differences? , 2014, Journal of Computational Physics.

[290]  Charles Audet,et al.  Pattern Search Algorithms for Mixed Variable Programming , 2000, SIAM J. Optim..

[291]  L. N. Vicente,et al.  Complexity and global rates of trust-region methods based on probabilistic models , 2018 .

[292]  Adam D. Bull,et al.  Convergence Rates of Efficient Global Optimization Algorithms , 2011, J. Mach. Learn. Res..

[293]  O. SIAMJ..  A DERIVATIVE-FREE ALGORITHM FOR INEQUALITY CONSTRAINED NONLINEAR PROGRAMMING VIA SMOOTHING OF AN ∞ PENALTY FUNCTION∗ , 2009 .

[294]  Luigi Grippo,et al.  Stopping criteria for linesearch methods without derivatives , 1984, Math. Program..

[295]  Christine A. Shoemaker,et al.  ORBIT: Optimization by Radial Basis Function Interpolation in Trust-Regions , 2008, SIAM J. Sci. Comput..

[296]  Qingna Li,et al.  A class of derivative-free methods for large-scale nonlinear monotone equations , 2011 .

[297]  Paul Belitz,et al.  New horizons in sphere-packing theory, part II: lattice-based derivative-free optimization via global surrogates , 2013, J. Glob. Optim..

[298]  A. L. Custódio,et al.  GLODS: Global and Local Optimization using Direct Search , 2014, Journal of Global Optimization.

[299]  Jeffrey C. Lagarias,et al.  Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions , 1998, SIAM J. Optim..

[300]  V. Torczon,et al.  Direct search methods: then and now , 2000 .

[301]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[302]  Fred W. Glover,et al.  Simulation optimization: a review, new developments, and applications , 2005, Proceedings of the Winter Simulation Conference, 2005..

[303]  A. Neumaier,et al.  A grid algorithm for bound constrained optimization of noisy functions , 1995 .

[304]  Katya Scheinberg,et al.  A derivative free optimization algorithm in practice , 1998 .

[305]  Achille Fokoue,et al.  An effective algorithm for hyperparameter optimization of neural networks , 2017, IBM J. Res. Dev..

[306]  Katya Scheinberg,et al.  Stochastic optimization using a trust-region method and random models , 2015, Mathematical Programming.

[307]  Alexander J. Smola,et al.  Stochastic Variance Reduction for Nonconvex Optimization , 2016, ICML.

[308]  Christine A. Shoemaker,et al.  Global Convergence of Radial Basis Function Trust Region Derivative-Free Algorithms , 2011, SIAM J. Optim..

[309]  Ronen Eldan,et al.  Bandit Smooth Convex Optimization: Improving the Bias-Variance Tradeoff , 2015, NIPS.

[310]  I. Coope,et al.  Frame Based Methods for Unconstrained Optimization , 2000 .

[311]  Aimo A. Törn,et al.  Global Optimization , 1999, Science.

[312]  M. Montaz Ali,et al.  A derivative-free algorithm for linearly constrained optimization problems , 2014, Comput. Optim. Appl..

[313]  Charles Audet,et al.  Derivative-Free and Blackbox Optimization , 2017 .

[314]  Warren Hare,et al.  Derivative-free optimization methods for finite minimax problems , 2013, Optim. Methods Softw..

[315]  Sébastien Bubeck,et al.  Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems , 2012, Found. Trends Mach. Learn..

[316]  Sivaraman Balakrishnan,et al.  Stochastic Zeroth-order Optimization in High Dimensions , 2017, AISTATS.

[317]  J. J. Moré,et al.  Levenberg--Marquardt algorithm: implementation and theory , 1977 .

[318]  R. L. McKinney.  Positive bases for linear spaces , 1962 .

[319]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[320]  J. M. Martínez,et al.  Derivative-free methods for nonlinear programming with general lower-level constraints , 2011 .

[321]  K. Madsen.  Minimax solution of non-linear equations without calculating derivatives , 1975 .

[322]  Christian L. Müller,et al.  Optimization of Convex Functions with Random Pursuit , 2011, SIAM J. Optim..

[323]  Hariharan Narayanan,et al.  Escaping the Local Minima via Simulated Annealing: Optimization of Approximately Convex Functions , 2015, COLT.

[324]  Michael I. Jordan,et al.  Fast Black-box Variational Inference through Stochastic Trust-Region Optimization , 2017, NIPS.

[325]  Philippe L. Toint,et al.  A derivative-free trust-funnel method for equality-constrained nonlinear optimization , 2014, Computational Optimization and Applications.

[326]  Xiaowei Hu,et al.  (Bandit) Convex Optimization with Biased Noisy Gradient Oracles , 2015, AISTATS.

[327]  Ishan Bajaj,et al.  A trust region-based two phase algorithm for constrained black-box and grey-box optimization with infeasible initial point , 2017, Comput. Chem. Eng..

[328]  J. M. Martínez,et al.  A derivative-free nonmonotone line-search technique for unconstrained optimization , 2008 .

[329]  Anne Auger,et al.  Experimental Comparisons of Derivative Free Optimization Algorithms , 2009, SEA.

[330]  Jianjun Gao,et al.  A polynomial case of convex integer quadratic programming problems with box integer constraints , 2015, J. Glob. Optim..

[331]  O. SIAMJ.,et al.  ON THE CONVERGENCE OF PATTERN SEARCH ALGORITHMS , 1997 .

[332]  Ky Khac Vu,et al.  Surrogate-based methods for black-box optimization , 2017, Int. Trans. Oper. Res..

[333]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[334]  P. Toint,et al.  Exploiting Band Structure in Unconstrained Optimization Without Derivatives , 2001 .

[335]  P. Toint,et al.  An Algorithm using Quadratic Interpolation for Unconstrained Derivative Free Optimization , 1996 .

[336]  Serge Gratton,et al.  An active-set trust-region method for derivative-free nonlinear bound-constrained optimization , 2011, Optim. Methods Softw..

[337]  Katya Scheinberg,et al.  On the construction of quadratic models for derivative-free trust-region algorithms , 2017, EURO J. Comput. Optim..

[338]  J. Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[339]  Shifeng Xiong,et al.  Sequential Design and Analysis of High-Accuracy and Low-Accuracy Computer Codes , 2013, Technometrics.

[340]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[341]  M. Powell.  A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation , 1994 .

[342]  Philippe L. Toint,et al.  BFO, A Trainable Derivative-free Brute Force Optimizer for Nonlinear Bound-constrained Optimization and Equilibrium Computations with Continuous and Discrete Variables , 2017, ACM Trans. Math. Softw..

[343]  Philippe L. Toint,et al.  A Derivative-Free Algorithm for Sparse Unconstrained Optimization Problems , 2002 .

[344]  Flagot Yohannes.  Derivative free optimization methods , 2012 .

[345]  Claude Brezinski,et al.  Numerical recipes in Fortran (The art of scientific computing) : W.H. Press, S.A. Teukolsky, W.T. Vetterling and B.P. Flannery, Cambridge Univ. Press, Cambridge, 2nd ed., 1992. 963 pp., US$49.95, ISBN 0-521-43064-X.☆ , 1993 .

[346]  Robert Michael Lewis,et al.  A Globally Convergent Augmented Lagrangian Pattern Search Algorithm for Optimization with General Constraints and Simple Bounds , 2002, SIAM J. Optim..

[347]  Jerrold H. May,et al.  Solving Nonlinear Programs Without Using Analytic Derivatives , 1979, Oper. Res..

[348]  Christine A. Shoemaker,et al.  MAPO: using a committee of algorithm-experts for parallel optimization of costly functions , 2003, SPAA '03.

[349]  Stefano Lucidi,et al.  Derivative-Free Methods for Mixed-Integer Constrained Optimization Problems , 2015, J. Optim. Theory Appl..

[350]  Jeffrey Larson,et al.  Stochastic derivative-free optimization using a trust region framework , 2016, Comput. Optim. Appl..

[351]  Trond Steihaug,et al.  On a New Method for Derivative Free Optimization , 2011 .

[352]  Stefan M. Wild,et al.  Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[353]  Charles Audet,et al.  A MADS Algorithm with a Progressive Barrier for Derivative-Free Nonlinear Programming , 2007 .

[354]  Martin J. Wainwright,et al.  Information-theoretic lower bounds on the oracle complexity of convex optimization , 2009, NIPS.

[355]  Shiyu Chang,et al.  Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization , 2018, NeurIPS.

[356]  Tamara G. Kolda,et al.  Asynchronous Parallel Pattern Search for Nonlinear Optimization , 2001, SIAM J. Sci. Comput..

[357]  M. J. D. Powell,et al.  On fast trust region methods for quadratic models with linear constraints , 2015, Math. Program. Comput..

[358]  Luís Nunes Vicente,et al.  Worst case complexity of direct search , 2013, EURO J. Comput. Optim..

[359]  Christian Kirches,et al.  Mixed-integer nonlinear optimization*† , 2013, Acta Numerica.

[360]  Kuo-Hao Chang,et al.  Stochastic Nelder-Mead simplex method - A new globally convergent direct search method for simulation optimization , 2012, Eur. J. Oper. Res..

[361]  Charles Audet,et al.  OrthoMADS: A Deterministic MADS Instance with Orthogonal Directions , 2008, SIAM J. Optim..

[362]  Paul Tseng,et al.  Gilding the Lily: A Variant of the Nelder-Mead Algorithm Based on Golden-Section Search , 2002, Comput. Optim. Appl..

[363]  D. Finkel,et al.  CONVERGENCE ANALYSIS OF SAMPLING METHODS FOR PERTURBED LIPSCHITZ FUNCTIONS , 2008 .

[364]  C. J. Price,et al.  Exploiting problem structure in pattern search methods for unconstrained optimization , 2006, Optim. Methods Softw..

[365]  Masha Sosonkina,et al.  Performance Modeling and Analysis of a Massively Parallel Direct—Part 1 , 2009, Int. J. High Perform. Comput. Appl..

[366]  M. J. D. Powell,et al.  On the use of quadratic models in unconstrained minimization without derivatives , 2004, Optim. Methods Softw..

[367]  Masha Sosonkina,et al.  Performance Modeling and Analysis of a Massively Parallel Direct—Part 2 , 2009, Int. J. High Perform. Comput. Appl..

[368]  Tamara G. Kolda,et al.  Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods , 2003, SIAM Rev..

[369]  Martin J. Wainwright,et al.  Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations , 2013, IEEE Transactions on Information Theory.

[370]  Marco Sciandrone,et al.  An implicit filtering algorithm for derivative-free multiobjective optimization with box constraints , 2017, Computational Optimization and Applications.

[371]  Robert D. Nowak,et al.  Query Complexity of Derivative-Free Optimization , 2012, NIPS.

[372]  Stefano Lucidi,et al.  Derivative-Free Robust Optimization for Circuit Design , 2015, J. Optim. Theory Appl..

[373]  L. Grippo,et al.  A class of derivative-free nonmonotone optimization algorithms employing coordinate rotations and gradient approximations , 2015, Comput. Optim. Appl..

[374]  Peter Auer,et al.  The Nonstochastic Multiarmed Bandit Problem , 2002, SIAM J. Comput..

[375]  Rémi Munos,et al.  Stochastic Simultaneous Optimistic Optimization , 2013, ICML.

[376]  Serge Gratton,et al.  A decoupled first/second-order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds , 2020, Math. Program..

[377]  C. Derman.  An Application of Chung's Lemma to the Kiefer-Wolfowitz Stochastic Approximation Procedure , 1956 .

[378]  Adrian S. Lewis,et al.  Estimating Tangent and Normal Cones Without Calculus , 2005, Math. Oper. Res..

[379]  Mark A. Abramson,et al.  Second-Order Behavior of Pattern Search , 2005, SIAM J. Optim..

[380]  P. Gill,et al.  Computing Forward-Difference Intervals for Numerical Optimization , 1983 .

[381]  Jorge J. Moré,et al.  Computing a Trust Region Step , 1983 .

[382]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[383]  William H. Press,et al.  Book-Review - Numerical Recipes in Pascal - the Art of Scientific Computing , 1989 .

[384]  M. J. Box.  A Comparison of Several Current Optimization Methods, and the use of Transformations in Constrained Problems , 1966, Comput. J..

[385]  Katya Scheinberg,et al.  Self-Correcting Geometry in Model-Based Algorithms for Derivative-Free Unconstrained Optimization , 2010, SIAM J. Optim..

[386]  Coralia Cartis,et al.  A derivative-free Gauss–Newton method , 2017, Mathematical Programming Computation.

[387]  Margaret H. Wright,et al.  Direct search methods: Once scorned, now respectable , 1996 .

[388]  Luís N. Vicente,et al.  Analysis of direct searches for discontinuous functions , 2012, Math. Program..

[389]  Mattias Björkman,et al.  Global Optimization of Costly Nonconvex Functions Using Radial Basis Functions , 2000 .

[390]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[391]  William La Cruz.  A projected derivative-free algorithm for nonlinear equations with convex constraints , 2014, Optim. Methods Softw..

[392]  K. I. M. McKinnon,et al.  Convergence of the Nelder-Mead Simplex Method to a Nonstationary Point , 1998, SIAM J. Optim..

[393]  Stefan M. Wild,et al.  Estimating Computational Noise , 2011, SIAM J. Sci. Comput..

[394]  Charles Audet,et al.  Analysis of Generalized Pattern Searches , 2000, SIAM J. Optim..

[395]  Juliane Müller,et al.  GOSAC: global optimization with surrogate approximation of constraints , 2017, J. Glob. Optim..

[396]  Katya Scheinberg,et al.  Geometry of interpolation sets in derivative free optimization , 2007, Math. Program..

[397]  V. G. Karmanov.  Convergence estimates for iterative minimization methods , 1974 .

[398]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[399]  W. Hare,et al.  Numerical Analysis of VU-Decomposition, U-Gradient, and U-Hessian Approximations , 2014, SIAM J. Optim..

[400]  Luís N. Vicente,et al.  On the optimal order of worst case complexity of direct search , 2016, Optim. Lett..

[401]  Paul Tseng,et al.  Objective-derivative-free methods for constrained optimization , 2002, Math. Program..

[402]  Owen J. Eslinger,et al.  Optimization of Automotive Valve Train Components with Implicit Filtering , 2000 .

[403]  D. Anderson,et al.  Algorithms for minimization without derivatives , 1974 .

[404]  M. Bierlaire,et al.  Boosters: A Derivative-Free Algorithm Based on Radial Basis Functions , 2009 .

[405]  Charles Audet,et al.  Parallel Space Decomposition of the Mesh Adaptive Direct Search Algorithm , 2007, SIAM J. Optim..

[406]  Alexander H. G. Rinnooy Kan,et al.  A stochastic method for global optimization , 1982, Math. Program..

[407]  Warren Hare,et al.  Derivative-Free Optimization Via Proximal Point Methods , 2014, J. Optim. Theory Appl..

[408]  Daniel John Woods,et al.  An interactive approach for solving multi-objective optimization problems (interactive computer, nelder-mead simplex algorithm, graphics) , 1985 .

[409]  Charles Audet,et al.  Robust optimization of noisy blackbox problems using the Mesh Adaptive Direct Search algorithm , 2016, Optim. Lett..

[410]  M. J. D. Powell,et al.  Direct search algorithms for optimization calculations , 1998, Acta Numerica.

[411]  Peter I. Frazier,et al.  A Tutorial on Bayesian Optimization , 2018, ArXiv.

[412]  W. Yu.  POSITIVE BASIS AND A CLASS OF DIRECT SEARCH TECHNIQUES , 1979 .

[413]  Robert L. Smith,et al.  Pure adaptive search in monte carlo optimization , 1989, Math. Program..

[414]  Ohad Shamir,et al.  On the Complexity of Bandit and Derivative-Free Stochastic Convex Optimization , 2012, COLT.

[415]  T. L. Lai Andherbertrobbins.  Asymptotically Efficient Adaptive Allocation Rules , 2022 .

[416]  L. N. Vicente,et al.  Smoothing and worst-case complexity for direct-search methods in nonsmooth optimization , 2013 .

[417]  Andreas Griewank,et al.  A mathematical view of automatic differentiation , 2003, Acta Numerica.

[418]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[419]  H. Kushner,et al.  RATES OF CONVERGENCE FOR STOCHASTIC APPROXIMATION TYPE ALGORITHMS , 1979 .

[420]  Tim Hesterberg,et al.  Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control , 2004, Technometrics.

[421]  K. Rashid,et al.  An adaptive multiquadric radial basis function method for expensive black-box mixed-integer nonlinear constrained optimization , 2013 .

[422]  Chandler Davis.  THEORY OF POSITIVE LINEAR DEPENDENCE. , 1954 .

[423]  Lin Xiao,et al.  Optimal Algorithms for Online Convex Optimization with Multi-Point Bandit Feedback. , 2010, COLT 2010.

[424]  Stefano Lucidi,et al.  A Derivative-Free Algorithm for Inequality Constrained Nonlinear Programming via Smoothing of an linfty Penalty Function , 2009, SIAM J. Optim..

[425]  Michael C. Ferris,et al.  Adaptation of the Uobyqa Algorithm for Noisy Functions , 2006, Proceedings of the 2006 Winter Simulation Conference.

[426]  Marco Sciandrone,et al.  On the Global Convergence of Derivative-Free Methods for Unconstrained Optimization , 2002, SIAM J. Optim..

[427]  Stephan Dempe,et al.  Direct search algorithm for bilevel programming problems , 2011, Comput. Optim. Appl..

[428]  M. J. D. Powell,et al.  An efficient method for finding the minimum of a function of several variables without calculating derivatives , 1964, Comput. J..

[429]  M. Powell.  A View of Algorithms for Optimization without Derivatives 1 , 2007 .

[430]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[431]  Stefano Lucidi,et al.  Derivative free methodologies for circuit worst case analysis , 2018, Optim. Lett..

[432]  Joshua D. Knowles,et al.  On metrics for comparing nondominated sets , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[433]  Mark A. Abramson,et al.  Pattern search ranking and selection algorithms for mixed variable simulation-based optimization , 2009, Eur. J. Oper. Res..

[434]  M. J. D. Powell,et al.  On trust region methods for unconstrained minimization without derivatives , 2003, Math. Program..

[435]  E. Omojokun.  Trust region algorithms for optimization with nonlinear equality and inequality constraints , 1990 .

[436]  Adil M. Bagirov,et al.  Piecewise Partially Separable Functions and a Derivative-free Algorithm for Large Scale Nonsmooth Optimization , 2006, J. Glob. Optim..

[437]  M. J. D. Powell,et al.  Least Frobenius norm updating of quadratic models that satisfy interpolation conditions , 2004, Math. Program..

[438]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[439]  Christine A. Shoemaker,et al.  SO-MI: A surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems , 2013, Comput. Oper. Res..

[440]  Stefan M. Wild.  MNH: A Derivative-Free Optimization Algorithm Using Minimal Norm Hessians , 2008 .

[441]  Eli Upfal,et al.  Multi-Armed Bandits in Metric Spaces ∗ , 2008 .

[442]  LiuzziGiampaolo,et al.  Derivative-Free Methods for Mixed-Integer Constrained Optimization Problems , 2015 .

[443]  Charles Audet,et al.  A Pattern Search Filter Method for Nonlinear Programming without Derivatives , 2001, SIAM J. Optim..

[444]  Jong-hyun Ryu,et al.  A Derivative-Free Trust-Region Method for Biobjective Optimization , 2014, SIAM J. Optim..

[445]  H. Robbins.  Some aspects of the sequential design of experiments , 1952 .

[446]  Jorge Nocedal,et al.  Derivative-Free Optimization of Noisy Functions via Quasi-Newton Methods , 2018, SIAM J. Optim..

[447]  Yurii Nesterov,et al.  Random Gradient-Free Minimization of Convex Functions , 2015, Foundations of Computational Mathematics.

[448]  Charles Audet,et al.  Efficient solution of quadratically constrained quadratic subproblems within the mesh adaptive direct search algorithm , 2017, Eur. J. Oper. Res..