Restarted Local Search Algorithms for Continuous Black Box Optimization

Several local search algorithms for real-valued domains (axis parallel line search, Nelder-Mead simplex search, Rosenbrock's algorithm, quasi-Newton method, NEWUOA, and VXQR) are described and thoroughly compared in this article, embedding them in a multi-start method. Their comparison aims (1) to help the researchers from the evolutionary community to choose the right opponent for their algorithm (to choose an opponent that would constitute a hard-to-beat baseline algorithm), (2) to describe individual features of these algorithms and show how they influence the algorithm on different problems, and (3) to provide inspiration for the hybridization of evolutionary algorithms with these local optimizers. The recently proposed Comparing Continuous Optimizers (COCO) methodology was adopted as the basis for the comparison. The results show that in low dimensional spaces, the old method of Nelder and Mead is still the most successful among those compared, while in spaces of higher dimensions, it is better to choose an algorithm based on quadratic modeling, such as NEWUOA or a quasi-Newton method.

[1]  Arnold Neumaier,et al.  Global Optimization by Multilevel Coordinate Search , 1999, J. Glob. Optim..

[2]  Pablo Moscato,et al.  On Evolution, Search, Optimization, Genetic Algorithms and Martial Arts : Towards Memetic Algorithms , 1989 .

[3]  Alex A. Freitas,et al.  Evolutionary Computation , 2002 .

[4]  M. Powell.  The NEWUOA software for unconstrained optimization without derivatives , 2006 .

[5]  K. Price.  Differential evolution vs. the functions of the 2/sup nd/ ICEO , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[6]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[7]  Roger J.-B. Wets,et al.  Minimization by Random Search Techniques , 1981, Math. Oper. Res..

[8]  Thomas Stützle,et al.  Evaluating Las Vegas Algorithms: Pitfalls and Remedies , 1998, UAI.

[9]  Arnold Neumaier,et al.  VXQR: derivative-free unconstrained optimization based on QR factorizations , 2011, Soft Comput..

[10]  Petr Posík,et al.  BBOB-benchmarking the Rosenbrock's local search algorithm , 2009, GECCO '09.

[11]  L. Darrell Whitley,et al.  Evaluating Evolutionary Algorithms , 1996, Artif. Intell..

[12]  Fei Peng,et al.  Population-Based Algorithm Portfolios for Numerical Optimization , 2010, IEEE Transactions on Evolutionary Computation.

[13]  K. I. M. McKinnon,et al.  Convergence of the Nelder-Mead Simplex Method to a Nonstationary Point , 1998, SIAM J. Optim..

[14]  Hans-Paul Schwefel,et al.  Evolution and Optimum Seeking: The Sixth Generation , 1993 .

[15]  H. H. Rosenbrock,et al.  An Automatic Method for Finding the Greatest or Least Value of a Function , 1960, Comput. J..

[16]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[17]  Raymond Ros,et al.  Benchmarking the NEWUOA on the BBOB-2009 function testbed , 2009, GECCO '09.

[18]  Raymond Ros,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup , 2009 .

[19]  Raymond Ros,et al.  Benchmarking the BFGS algorithm on the BBOB-2009 function testbed , 2009, GECCO '09.

[20]  Edmund K. Burke,et al.  The Genetic and Evolutionary Computation Conference , 2011 .

[21]  Petr Posík,et al.  BBOB-benchmarking two variants of the line-search algorithm , 2009, GECCO '09.

[22]  Hugues Bersini,et al.  S.T.E.P.: the easiest way to optimize a function , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[23]  Anne Auger,et al.  Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 , 2010, GECCO '10.

[24]  J. R. Palmer.  An Improved Procedure for Orthogonalising the Search Vectors in Rosenbrock's and Swann's Direct Search Optimisation Methods , 1969, Comput. J..

[25]  Anne Auger,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions , 2009 .

[26]  Nikolaus Hansen,et al.  Benchmarking the nelder-mead downhill simplex algorithm with many local restarts , 2009, GECCO '09.

[27]  Hans-Paul Schwefel,et al.  Evolution and optimum seeking , 1995, Sixth-generation computer technology series.

[28]  Pierre Hansen,et al.  Variable Neighborhood Search , 2018, Handbook of Heuristics.