Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods

Direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives. Direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis. Nonetheless, users remained loyal to these methods, most of which were easy to program, some of which were reliable. In the past fifteen years, these methods have seen a revival due, in part, to the appearance of mathematical analysis, as well as to interest in parallel and distributed com- puting. This review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited. Our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results. The underlying principles allow general- ization to handle bound constraints and linear constraints. We also discuss extensions to problems with nonlinear constraints.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  G. Box,et al.  On the Experimental Attainment of Optimum Conditions , 1951 .

[3]  Chandler Davis.  THEORY OF POSITIVE LINEAR DEPENDENCE. , 1954 .

[4]  G. Box.  The Exploration and Exploitation of Response Surfaces: Some General Considerations and Examples , 1954 .

[5]  O. L. Davies,et al.  Design and analysis of industrial experiments , 1954 .

[6]  W. Davidon,et al.  SCATTERING OF POSITIVE PIONS BY HYDROGEN AT 189 MEV , 1955 .

[7]  J. Kiefer.  Optimum Sequential Search and Approximation Methods Under Minimum Regularity Assumptions , 1957 .

[8]  George E. P. Box,et al.  Evolutionary Operation: a Method for Increasing Industrial Productivity , 1957 .

[9]  Samuel H. Brooks.  A Comparison of Maximum-Seeking Methods , 1959 .

[10]  H. H. Rosenbrock,et al.  An Automatic Method for Finding the Greatest or Least Value of a Function , 1960, Comput. J..

[11]  Charles W. Carroll.  The Created Response Surface Technique for Optimizing Nonlinear, Restrained Systems , 1961 .

[12]  Samuel H. Brooks,et al.  Optimum Estimation of Gradient Direction in Steepest Ascent Experiments , 1961 .

[13]  Robert Hooke,et al.  `` Direct Search'' Solution of Numerical and Statistical Problems , 1961, JACM.

[14]  H. Spang.  A Review of Minimization Techniques for Nonlinear Functions , 1962 .

[15]  Raymond A. Mugele,et al.  A Program for Optimal Control of Nonlinear Processes , 1962, IBM Syst. J..

[16]  G. R. Hext,et al.  Sequential Application of Simplex Designs in Optimisation and Evolutionary Operation , 1962 .

[17]  A. Kaupe.  Algorithm 178: direct search , 1963, CACM.

[18]  Patrick D. Krolak,et al.  An extension of Fibonaccian search to several variables , 1963, CACM.

[19]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[20]  M. J. D. Powell,et al.  An efficient method for finding the minimum of a function of several variables without calculating derivatives , 1964, Comput. J..

[21]  M. Thomas,et al.  On "An extension of Fibonaccian searching to multidimensional cases" , 1964 .

[22]  W. R. Klingman,et al.  Nonlinear Programming with the Aid of a Multiple-Gradient Summation Technique , 1964, JACM.

[23]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[24]  R. Fletcher,et al.  Function Minimization Without Evaluating Derivatives - a Review , 1965, Comput. J..

[25]  M. J. D. Powell,et al.  A Method for Minimizing a Sum of Squares of Non-Linear Functions Without Calculating Derivatives , 1965, Comput. J..

[26]  M. J. Box.  A New Method of Constrained Optimization and a Comparison With Other Methods , 1965, Comput. J..

[27]  L. Cooper,et al.  Sequential Search: A Method for Solving Constrained Optimization Problems , 1965, JACM.

[28]  C. G. Broyden.  A Class of Methods for Solving Nonlinear Simultaneous Equations , 1965 .

[29]  Cyrus F. Wood,et al.  Review of Design Optimization Techniques , 1965, IEEE Trans. Syst. Sci. Cybern..

[30]  F. L. Chernous’ko.  A local variation method for the numerical solution of variational problems , 1965 .

[31]  I. D. Hill,et al.  Algorithm 291: logarithm of gamma function , 1966, CACM.

[32]  F. E. Emery,et al.  Optimal Design of Matching Networks for Microwave Transistor Amplifiers , 1966 .

[33]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .

[34]  G. Berman.  Minimization by Successive Approximation , 1966 .

[35]  M. J. Box.  A Comparison of Several Current Optimization Methods, and the use of Transformations in Constrained Problems , 1966, Comput. J..

[36]  Willard I. Zangwill,et al.  Minimizing a function without calculating derivatives , 1967, Comput. J..

[37]  Allen A. Goldstein,et al.  Constructive Real Analysis , 1967 .

[38]  R. A. Mitchell,et al.  Nonlinear constrained optimization by a nonrandom complex method , 1968 .

[39]  René de Vogelaere,et al.  Remark on algorithm 178: Direct search , 1968, Commun. ACM.

[40]  J. A. Guin,et al.  Modification of the Complex Method of Constrained Optimization , 1968, Comput. J..

[41]  M. J. D. Powell,et al.  Nonlinear Programming—Sequential Unconstrained Minimization Techniques , 1969 .

[42]  M. Powell.  A method for nonlinear constraints in minimization problems , 1969 .

[43]  M. J. Box,et al.  Non-linear optimization techniques; , 1969 .

[44]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[45]  J. R. Palmer.  An Improved Procedure for Orthogonalising the Search Vectors in Rosenbrock's and Swann's Direct Search Optimisation Methods , 1969, Comput. J..

[46]  Gerald Berman,et al.  Lattice Approximations to the Minima of Functions of Several Variables , 1969, JACM.

[47]  N. S. Barnett,et al.  Private communication , 1969 .

[48]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[49]  David M. Himmelblau,et al.  Constrained Nonlinear Optimization by Heuristic Programming , 1969, Oper. Res..

[50]  George E. P. Box,et al.  Evolutionary Operation: A Statistical Method for Process Improvement , 1969 .

[51]  A. Jones,et al.  Spiral - A New Algorithm for Non-Linear Parameter Estimation Using Least Squares , 1970, Comput. J..

[52]  G. Peckham,et al.  A New Method for Minimising a Sum of Squares Without Calculating Gradients , 1970, Comput. J..

[53]  W. Miranker,et al.  A Nongradient and Parallel Algorithm for Unconstrained Minimization , 1970 .

[54]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[55]  J. Cea.  Optimisation : théorie et algorithmes , 1971 .

[56]  E. H. Zarantonello.  Projections on Convex Sets in Hilbert Space and Spectral Theory: Part I. Projections on Convex Sets: Part II. Spectral Theory , 1971 .

[57]  A. Ichikawa,et al.  A Modified Complex Method for Optimization , 1971 .

[58]  R. O'Neill.  Algorithm AS 47: Function Minimization Using a Simplex Procedure , 1971 .

[59]  F. A. Lootsma,et al.  Numerical methods for non-linear optimization , 1974 .

[60]  anonymous.  A new algorithm for optimization , 1972 .

[61]  David Mautner Himmelblau,et al.  Applied Nonlinear Programming , 1972 .

[62]  J. Lawrence,et al.  An adaptive randomized pattern search , 1972, CDC 1972.

[63]  E. Polak,et al.  Computational methods in optimization : a unified approach , 1972 .

[64]  K. Pinder,et al.  Optimization of a Simulation Model of a Chemical Plant , 1972 .

[65]  L. C. W. Dixon,et al.  ASCIM - an accelerated constrained simplex technique , 1973, Comput. Aided Des..

[66]  R. Glass,et al.  REFLEX Method for Empirical Optimization , 1973 .

[67]  M. J. D. Powell,et al.  On search directions for minimization algorithms , 1973, Math. Program..

[68]  T. H. I. Jaakola,et al.  Optimization by direct search and systematic reduction of the size of search region , 1973 .

[69]  D. Winfield,et al.  Function Minimization by Interpolation in a Data Table , 1973 .

[70]  D. Keefer.  Simpat: Self-Bounding Direct Search Method for Optimization , 1973 .

[71]  S. Deming,et al.  UNIPLEX. Single-factor optimization of response in the presence of error , 1974 .

[72]  D. Anderson,et al.  Algorithms for minimization without derivatives , 1974 .

[73]  D. A. Phillips,et al.  A Preliminary Investigation of Function Optimisation by a Combination of Methods , 1974, Comput. J..

[74]  S. Deming,et al.  Simplex optimization of analytical chemical methods , 1974 .

[75]  S. Deming,et al.  Selection and preprocessing of factors for simplex optimization , 1974 .

[76]  J. E. Ertel,et al.  Remark AS R11: A Remark on Algorithm AS 47 "Function Minimization using a Simplex Procedure" , 1974 .

[77]  Robert Mifflin,et al.  A superlinearly convergent algorithm for minimization without evaluating derivatives , 1975, Math. Program..

[78]  K. Brodlie.  A new direction set method for unconstrained minimization without evaluating derivatives , 1975 .

[79]  L. S. Nelson,et al.  The Nelder-Mead Simplex Procedure for Function Minimization , 1975 .

[80]  M. J. D. Powell,et al.  A View of Unconstrained Minimization Algorithms that Do Not Require Derivatives , 1975, TOMS.

[81]  P. Benyon.  Remark AS R15: Function Minimization Using a Simplex Procedure , 1976 .

[82]  P. Boggs,et al.  A stability analysis for perturbed nonlinear iterative methods , 1976 .

[83]  P. Toint,et al.  On the accelerating property of an algorithm for function minimization without calculating derivatives , 1977 .

[84]  Wyn L. Price,et al.  A Controlled Random Search Procedure for Global Optimisation , 1977, Comput. J..

[85]  A. Goldstein,et al.  Optimization of functions whose values are subject to small errors , 1977 .

[86]  J. W. Akitt.  Function Minimisation Using the Nelder and Mead Simplex Method with Limited Arithmetic Precision: The Self Regenerative Simplex , 1977, Comput. J..

[87]  M. B. Denton,et al.  Performance of the Super Modified Simplex , 1977 .

[88]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[89]  P. Toint,et al.  On the accelerating property of an algorithm for function minimization without calculating derivatives , 1978 .

[90]  M. B. Denton,et al.  A Review of Simplex Optimization in Analytical Chemistry , 1978 .

[91]  I. D. Hill.  Remark AS R28: A Remark on Algorithm AS 47: Function Minimization Using a Simplex Procedure , 1978 .

[92]  James W. Evans.  Computer Augmentation of Experimental Designs to Maximize |X'X , 1979 .

[93]  W. Yu.  POSITIVE BASIS AND A CLASS OF DIRECT SEARCH TECHNIQUES , 1979 .

[94]  R. B. Spencer,et al.  High-speed algorithm for simplex optimization calculations , 1979 .

[95]  P. Ryan,et al.  Simplex techniques for nonlinear optimization , 1980 .

[96]  J. Dunn.  Global and Asymptotic Convergence Rate Estimates for a Class of Projected Gradient Processes , 1981 .

[97]  D. Walmsley.  THE SIMPLEX METHOD FOR MINIMISATION OF A GENERAL FUNCTION , 1981 .

[98]  Philip E. Gill,et al.  Practical optimization , 1981 .

[99]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[100]  A. Gustavsson,et al.  Design and evaluation of modified simplex methods , 1982 .

[101]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[102]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[103]  G. Kateman,et al.  The symmetry-controlled simplex optimization procedure , 1983 .

[104]  Luigi Grippo,et al.  Stopping criteria for linesearch methods without derivatives , 1984, Math. Program..

[105]  Matts Roos,et al.  MINUIT-a system for function minimization and analysis of the parameter errors and correlations , 1984 .

[106]  O. Wing,et al.  Pseudo-conjugate directions for the solution of the nonlinear unconstrained optimization problem on a parallel computer , 1984 .

[107]  A. Gustavsson,et al.  Design and optimization of modified simplex methods , 1985 .

[108]  Daniel John Woods,et al.  An interactive approach for solving multi-objective optimization problems (interactive computer, nelder-mead simplex algorithm, graphics) , 1985 .

[109]  L. R. Parker,et al.  Comparison of simplex algorithms , 1985 .

[110]  Daniel J. Woods,et al.  Optimization on Microcomputers: The Nelder-Mead Simplex Algorithm , 1985 .

[111]  D. Betteridge,et al.  Reflections on the modified simplex-II. , 1985, Talanta.

[112]  D. Betteridge,et al.  Reflections on the modified simplex-I. , 1985, Talanta.

[113]  M. Cave.  An improved skmplex algorithm for dealing with boundary conditions , 1986 .

[114]  D. Chen,et al.  A New Simplex Procedure For Function Minimization , 1986 .

[115]  David K. Smith,et al.  A minimization solver for many parameters , 1986 .

[116]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[117]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[118]  K. Burton,et al.  Optimisation via simplex , 1987 .

[119]  P. Toint,et al.  Global convergence of a class of trust region algorithms for optimization with simple bounds , 1988 .

[120]  L. Grippo,et al.  Global convergence and stabilization of unconstrained minimization methods without derivatives , 1988 .

[121]  Griewank,et al.  On automatic differentiation , 1988 .

[122]  V. J. Torczoit,et al.  Multidirectional search: a direct search algorithm for parallel machines , 1989 .

[123]  J. Strikwerda.  Finite Difference Schemes and Partial Differential Equations , 1989 .

[124]  John E. Dennis,et al.  Multidirectional search: a direct search algorithm for parallel machines , 1989 .

[125]  Robert Mifflin,et al.  A bracketing technique to ensure desirable convergence in univariate minimization , 1989, Math. Program..

[126]  T. Rowan.  Functional stability analysis of numerical algorithms , 1990 .

[127]  K. Burton,et al.  Optimization using the super-modified simplex method , 1990 .

[128]  P. Fajer,et al.  General method for multiparameter fitting of high-resolution EPR spectra using a simplex algorithm☆ , 1990 .

[129]  K. Burton,et al.  Optimization using the modified simplex method , 1990 .

[130]  John E. Dennis,et al.  Direct Search Methods on Parallel Machines , 1991, SIAM J. Optim..

[131]  William C. Davidon,et al.  Variable Metric Method for Minimization , 1959, SIAM J. Optim..

[132]  P. Toint,et al.  A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds , 1991 .

[133]  Shirley Dex,et al.  JR 旅客販売総合システム（マルス）における運用及び管理について , 1991 .

[134]  M. Shao.  Techniques and performance of the advanced regular simplex method , 1991 .

[135]  David Avis,et al.  A pivoting algorithm for convex hulls and vertex enumeration of arrangements and polyhedra , 1991, SCG '91.

[136]  Virginia Torczon,et al.  On the Convergence of the Multidirectional Search Algorithm , 1991, SIAM J. Optim..

[137]  Andreas Griewank,et al.  ADIFOR - Generating Derivative Codes form Fortran Programs , 1992, Sci. Program..

[138]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[139]  Peter Hedlund,et al.  Design and evaluation of modified simplex methods having enhanced convergence ability , 1992 .

[140]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[141]  Robert Mifflin,et al.  A rapidly convergent five-point algorithm for univariate minimization , 1993, Math. Program..

[142]  S. K. Zavriev,et al.  On the global optimization properties of finite-difference local descent algorithms , 1993, J. Glob. Optim..

[143]  Brian W. Kernighan,et al.  AMPL: A Modeling Language for Mathematical Programming , 1993 .

[144]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[145]  M. Powell.  A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation , 1994 .

[146]  P. GILMOREy,et al.  Implicit Filtering and Optimal Design Problems , 1994 .

[147]  Carolyn Pillers Dobler.  A Matrix Approach to Finding a Set of Generators and Finding the Polar (Dual) of a Class of Polyhedral Cones , 1994 .

[148]  Paul D. Hovland,et al.  ADIFOR 2.0 user`s guide (Revision B) , 1995 .

[149]  M. Berthod,et al.  Direct Search Generalized Simplex Algorithm for Optimizing Non-linear Functions , 1995 .

[150]  Cass T. Miller,et al.  Implicit Filtering and Optimal Design Problems , 1995 .

[151]  C. T. Kelley,et al.  An Implicit Filtering Algorithm for Optimization of Functions with Many Local Minima , 1995, SIAM J. Optim..

[152]  A. Neumaier,et al.  A grid algorithm for bound constrained optimization of noisy functions , 1995 .

[153]  Roland Glowinski,et al.  On the Simulation and Control of Some Friction Constrained Motions , 1995, SIAM J. Optim..

[154]  William E. Hart,et al.  Evolutionary pattern search algorithms , 1995 .

[155]  William E. Hart,et al.  A Stationary Point Convergence Theory for Evolutionary Algorithms , 1996, FOGA.

[156]  V. Torczon,et al.  RANK ORDERING AND POSITIVE BASES IN PATTERN SEARCH ALGORITHMS , 1996 .

[157]  Lucidi Stefano,et al.  Numerical Results for Unconstrained Optimization without Derivatives , 1996 .

[158]  Juan C. Meza,et al.  A comparison of a direct search method and a genetic algorithm for conformational searching , 1996, J. Comput. Chem..

[159]  P. Toint,et al.  An Algorithm using Quadratic Interpolation for Unconstrained Derivative Free Optimization , 1996 .

[160]  Margaret H. Wright,et al.  Direct search methods: Once scorned, now respectable , 1996 .

[161]  Michael W. Trosset,et al.  I Know It When I See It: Toward a Definition of Direct Search Methods , 1996 .

[162]  Anthony J. Kearsley.  The use of optimization techniques in the solution of partial differential equations from science and engineering , 1996 .

[163]  Natalia Alexandrov,et al.  Multidisciplinary design optimization : state of the art , 1997 .

[164]  Katya Scheinberg,et al.  Recent progress in unconstrained nonlinear optimization without derivatives , 1997, Math. Program..

[165]  Virginia Torczon,et al.  Numerical Optimization Using Computer Experiments , 1997 .

[166]  P. A. Macdonald,et al.  Optimization of Microwave Networks by Razor Search , 1969 .

[167]  O. SIAMJ.,et al.  ON THE CONVERGENCE OF PATTERN SEARCH ALGORITHMS , 1997 .

[168]  Christian H. Bischof,et al.  ADIC: an extensible automatic differentiation tool for ANSI‐C , 1997, Softw. Pract. Exp..

[169]  Katya Scheinberg,et al.  On the convergence of derivative-free methods for unconstrained optimization , 1997 .

[170]  William E. Hart,et al.  A Generalized Stationary Point Convergence Theory for Evolutionary Algorithms , 1997, ICGA.

[171]  Christian Bischof,et al.  The ADIFOR 2.0 system for the automatic differentiation of Fortran 77 programs , 1997 .

[172]  C. Kelley,et al.  The Simplex Gradient and Noisy Optimization Problems , 1998 .

[173]  V. Torczon,et al.  From Evolutionary Operation to Parallel Direct Search : Pattern Search Algorithms for Numerical Optimization , 1998 .

[174]  V. Torczon,et al.  Why Pattern Search Works , 1998 .

[175]  John E. Dennis,et al.  Optimization Using Surrogate Objectives on a Helicopter Test Example , 1998 .

[176]  Ya-Xiang Yuan,et al.  An Example of Non-Convergence of Trust Region Algorithms , 1998 .

[177]  T. Öberg.  Importance of the first design matrix in experimental simplex optimization , 1998 .

[178]  Katya Scheinberg,et al.  A derivative free optimization algorithm in practice , 1998 .

[179]  Jeffrey C. Lagarias,et al.  Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions , 1998, SIAM J. Optim..

[180]  K. I. M. McKinnon,et al.  Convergence of the Nelder-Mead Simplex Method to a Nonstationary Point , 1998, SIAM J. Optim..

[181]  M. J. D. Powell,et al.  Direct search algorithms for optimization calculations , 1998, Acta Numerica.

[182]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[183]  Robert Michael Lewis,et al.  Pattern Search Algorithms for Bound Constrained Minimization , 1999, SIAM J. Optim..

[184]  Paul Tseng,et al.  Fortified-Descent Simplicial Search Method: A General Approach , 1999, SIAM J. Optim..

[185]  C. T. Kelley,et al.  Detection and Remediation of Stagnation in the Nelder--Mead Algorithm Using a Sufficient Decrease Condition , 1999, SIAM J. Optim..

[186]  Carl Tim Kelley,et al.  Iterative methods for optimization , 1999, Frontiers in applied mathematics.

[187]  T. Coleman,et al.  Siag/opt Views-and-news a Forum for the Siam Activity Group on Optimization Chairman's Column Case Studies Ppcx: Solving Linear Programs in Parallel , 2007 .

[188]  C. T. Kelley,et al.  Superlinear Convergence and Implicit Filtering , 1999, SIAM J. Optim..

[189]  Robert Michael Lewis,et al.  Pattern Search Methods for Linearly Constrained Minimization , 1999, SIAM J. Optim..

[190]  Adam P. Gurson,et al.  Simplex Search Behavior in Nonlinear Optimization , 2000 .

[191]  Christopher M. Siefert,et al.  Model-Assisted Pattern Search , 2000 .

[192]  Jose Rodriguez,et al.  Second-order information in the adaptive search exploration algorithm , 2000 .

[193]  I. D. Coope,et al.  A direct search conjugate directions algorithm for unconstrained minimization , 2000 .

[194]  M. Trosset.  On the Use of Direct Search Methods for Stochastic Optimization , 2000 .

[195]  Aiaa Oo.  Institute for Computer Applications in Science , 2000 .

[196]  I. Coope,et al.  Frame Based Methods for Unconstrained Optimization , 2000 .

[197]  Margaret H. Wright,et al.  What, if anything, is new in optimization? , 2000 .

[198]  Stephen E. Wright,et al.  A Note on Positively Spanning Sets , 2000, Am. Math. Mon..

[199]  V. Torczon,et al.  Direct search methods: then and now , 2000 .

[200]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[201]  Tamara G. Kolda,et al.  Asynchronous Parallel Pattern Search for Nonlinear Optimization , 2001, SIAM J. Sci. Comput..

[202]  C. J. Price,et al.  On the Convergence of Grid-Based Methods for Unconstrained Optimization , 2000, SIAM J. Optim..

[203]  P. Tseng.  Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization , 2001 .

[204]  M. Powell.  On the Lagrange functions of quadratic models that are defined by interpolation* , 2001 .

[205]  J. Dennis,et al.  Mixed Variable Optimization of the Number and Composition of Heat Intercepts in a Thermal Insulation System , 2001 .

[206]  Charles Audet,et al.  Pattern Search Algorithms for Mixed Variable Programming , 2000, SIAM J. Optim..

[207]  C. T. Kelley,et al.  A Locally-Biased form of the DIRECT Algorithm , 2001, J. Glob. Optim..

[208]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[209]  Michael C. Ferris,et al.  A Direct Search Algorithm for Optimization with Noisy Function Evaluations , 2000, SIAM J. Optim..

[210]  Robert Michael Lewis,et al.  A Globally Convergent Augmented Lagrangian Pattern Search Algorithm for Optimization with General Constraints and Simple Bounds , 2002, SIAM J. Optim..

[211]  Marco Sciandrone,et al.  A Derivative-Free Algorithm for Bound Constrained Optimization , 2002, Comput. Optim. Appl..

[212]  Marco Sciandrone,et al.  On the Global Convergence of Derivative-Free Methods for Unconstrained Optimization , 2002, SIAM J. Optim..

[213]  Juan C. Meza,et al.  A Class of Trust-Region Methods for Parallel Optimization , 1999, SIAM J. Optim..

[214]  Paul Tseng,et al.  Gilding the Lily: A Variant of the Nelder-Mead Algorithm Based on Golden-Section Search , 2002, Comput. Optim. Appl..

[215]  Sven Leyffer,et al.  Nonlinear programming without a penalty function , 2002, Math. Program..

[216]  Sven Leyffer,et al.  On the Global Convergence of a Filter--SQP Algorithm , 2002, SIAM J. Optim..

[217]  Ubaldo M. García-Palomares,et al.  New Sequential and Parallel Derivative-Free Algorithms for Unconstrained Minimization , 2002, SIAM J. Optim..

[218]  Charles Audet,et al.  Analysis of Generalized Pattern Searches , 2000, SIAM J. Optim..

[219]  C. J. Price,et al.  Positive Bases in Numerical Optimization , 2002, Comput. Optim. Appl..

[220]  I. D. Coope,et al.  A Convergent Variant of the Nelder–Mead Algorithm , 2002 .

[221]  J. Burkardt,et al.  Insensitive Functionals, Inconsistent Gradients, Spurious Minima, and Regularized Functionals in Flow Optimization Problems , 2002 .

[222]  Dominique Pelletier,et al.  On Sensitivity Analysis for Problems with Numerical Noise , 2002 .

[223]  Jorge Nocedal,et al.  Wedge trust region methods for derivative free optimization , 2002, Math. Program..

[224]  Charles Audet,et al.  Convergence Results for Pattern Search Algorithms are Tight , 2002 .

[225]  Paul Tseng,et al.  Objective-derivative-free methods for constrained optimization , 2002, Math. Program..

[226]  I. Coope,et al.  Frame-Based Ray Search Algorithms in Unconstrained Optimization , 2003 .

[227]  Michael Wetter,et al.  Generalized pattern search algorithms with adaptive precision function evaluations , 2003 .

[228]  Tamara G. Kolda,et al.  Understanding Asynchronous Parallel Pattern Search , 2003 .

[229]  J. Dennis,et al.  Pattern search algorithms for mixed variable general constrained optimization problems , 2003 .

[230]  Charles Audet,et al.  Generalized pattern searches with derivative information , 2002, Math. Program..

[231]  Tamara G. Kolda,et al.  On the Convergence of Asynchronous Parallel Pattern Search , 2002, SIAM J. Optim..

[232]  Fernando Nogueira,et al.  Pattern Search Methods for User-Provided Points: Application to Molecular Geometry Problems , 2004, SIAM J. Optim..

[233]  Charles Audet,et al.  A Pattern Search Filter Method for Nonlinear Programming without Derivatives , 2001, SIAM J. Optim..