MSO: a framework for bound-constrained black-box global optimization algorithms

This paper addresses a class of algorithms for solving bound-constrained black-box global optimization problems. These algorithms partition the objective function domain over multiple scales in search for the global optimum. For such algorithms, we provide a generic procedure and refer to as multi-scale optimization (MSO). Furthermore, we propose a theoretical methodology to study the convergence of MSO algorithms based on three basic assumptions: (a) local Hölder continuity of the objective function f, (b) partitions boundedness, and (c) partitions sphericity. Moreover, the worst-case finite-time performance and convergence rate of several leading MSO algorithms, namely, Lipschitzian optimization methods, multi-level coordinate search, dividing rectangles, and optimistic optimization methods have been presented.

[1]  Philippe Preux,et al.  Bandits attack function optimization , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[2]  W. R. Thompson.  ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES , 1933 .

[3]  B. Shubert.  A Sequential Method Seeking the Global Maximum of a Function , 1972 .

[4]  D. Mayne,et al.  Outer approximation algorithm for nondifferentiable optimization problems , 1984 .

[5]  H. Robbins.  Some aspects of the sequential design of experiments , 1952 .

[6]  Laurence A. Wolsey,et al.  Integer and Combinatorial Optimization , 1988 .

[7]  Bruno Bouzy,et al.  Monte-Carlo strategies for computer Go , 2006 .

[8]  C. T. Kelley,et al.  A Locally-Biased form of the DIRECT Algorithm , 2001, J. Glob. Optim..

[9]  Clara Pizzuti,et al.  Local tuning and partition strategies for diagonal GO methods , 2003, Numerische Mathematik.

[10]  T. Reiland,et al.  Nonsmooth Analysis and Optimization for a Class of Nonconvex Mappings , 1985 .

[11]  R. G. Strongin,et al.  A global minimization algorithm with parallel iterations , 1990 .

[12]  Bilel Derbel,et al.  Simultaneous optimistic optimization on the noiseless BBOB testbed , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[13]  Rémi Munos,et al.  Optimistic Optimization of Deterministic Functions , 2011, NIPS 2011.

[14]  Y. Sergeyev.  On convergence of "divide the best" global optimization algorithms , 1998 .

[15]  D. Finkel,et al.  Convergence analysis of the direct algorithm , 2004 .

[16]  Qunfeng Liu,et al.  A modified DIRECT algorithm with bilevel partition , 2014, J. Glob. Optim..

[17]  Arnold Neumaier,et al.  Global Optimization by Multilevel Coordinate Search , 1999, J. Glob. Optim..

[18]  Rémi Munos,et al.  Stochastic Simultaneous Optimistic Optimization , 2013, ICML.

[19]  Tibor Csendes,et al.  On the selection of subdivision directions in interval branch-and-bound methods for global optimization , 1995, J. Glob. Optim..

[20]  Remigijus Paulavičius,et al.  Simplicial Global Optimization , 2014 .

[21]  Nando de Freitas,et al.  Bayesian Multi-Scale Optimistic Optimization , 2014, AISTATS.

[22]  Y. Evtushenko.  Numerical methods for finding global extrema (Case of a non-uniform mesh) , 1971 .

[23]  Julius Zilinskas,et al.  Globally-biased Disimpl algorithm for expensive global optimization , 2014, Journal of Global Optimization.

[24]  László Pál,et al.  A Comparison of Global Search Algorithms for Continuous Black Box Optimization , 2012, Evolutionary Computation.

[25]  N. Hansen,et al.  Real-Parameter Black-Box Optimization Benchmarking: Experimental Setup , 2010 .

[26]  János D. Pintér,et al.  Globally convergent methods for n-dimensional multiextremal optimization , 1986 .

[27]  A. G. Sukharev.  Optimal strategies of the search for an extremum , 1971 .

[28]  R. Horst,et al.  On the convergence of global methods in multiextremal optimization , 1987 .

[29]  Andreas Krause,et al.  Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting , 2009, IEEE Transactions on Information Theory.

[30]  T. L. Lai Andherbertrobbins.  Asymptotically Efficient Adaptive Allocation Rules , 1985 .

[31]  Roman G. Strongin,et al.  Global multidimensional optimization on parallel computer , 1992, Parallel Comput..

[32]  Yaroslav D. Sergeyev,et al.  Deterministic approaches for solving practical black-box global optimization problems , 2015, Adv. Eng. Softw..

[33]  C. T. Kelley,et al.  Modifications of the direct algorithm , 2001 .

[34]  J. M. Gablonsky.  An implementation of the direct algorithm , 1998 .

[35]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[36]  C. Floudas.  Handbook of Test Problems in Local and Global Optimization , 1999 .

[37]  J D Pinter,et al.  Global Optimization in Action—Continuous and Lipschitz Optimization: Algorithms, Implementations and Applications , 2010 .

[38]  Yaroslav D. Sergeyev,et al.  Global Search Based on Efficient Diagonal Partitions and a Set of Lipschitz Constants , 2006, SIAM J. Optim..

[39]  V. U. Malkova,et al.  Parallel global optimization of functions of several variables , 2009 .

[40]  S. A. Piyavskii.  An algorithm for finding the absolute extremum of a function , 1972 .

[41]  Ofer M. Shir,et al.  Accelerated optimization and automated discovery with covariance matrix adaptation for experimental quantum control , 2009 .

[42]  Yaroslav D. Sergeyev,et al.  A deterministic global optimization using smooth diagonal auxiliary functions , 2015, Commun. Nonlinear Sci. Numer. Simul..

[43]  Pierre Hansen,et al.  On the Number of Iterations of Piyavskii's Global Optimization Algorithm , 1991, Math. Oper. Res..

[44]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[45]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[46]  T. Csendes,et al.  A review of subdivision direction selection in interval methods for global optimization , 1997 .

[47]  Yaroslav D. Sergeyev,et al.  Lipschitz gradients for global optimization in a one-point-based partitioning scheme , 2012, J. Comput. Appl. Math..

[48]  Petr Posík,et al.  BBOB-benchmarking the DIRECT global optimization algorithm , 2009, GECCO '09.

[49]  Y. Sergeyev.  A one-dimensional deterministic global minimization algorithm , 1995 .

[50]  Daniel E. Finkel,et al.  Additive Scaling and the DIRECT Algorithm , 2006, J. Glob. Optim..

[51]  Victor V. Ivanov.  On Optimal Algorithms of Control , 1972 .

[52]  Aimo A. Törn,et al.  Global Optimization , 1999, Science.

[53]  Regina Hunter Mladineo.  An algorithm for finding the global maximum of a multimodal, multivariate function , 1986, Math. Program..

[54]  Mikhail Posypkin,et al.  A deterministic approach to global box-constrained optimization , 2012, Optimization Letters.

[55]  Yongqiang Wang,et al.  A Survey of Some Model-Based Methods for Global Optimization , 2012 .

[56]  Anne Auger,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions , 2009 .

[57]  Y. D. Sergeyev,et al.  Global Optimization with Non-Convex Constraints - Sequential and Parallel Algorithms (Nonconvex Optimization and its Applications Volume 45) (Nonconvex Optimization and Its Applications) , 2000 .

[58]  Nicholas I. M. Gould,et al.  A branch and bound algorithm for the global optimization of Hessian Lipschitz continuous functions , 2013, J. Glob. Optim..

[59]  Benjamin Perret,et al.  Constructive Links between Some Morphological Hierarchies on Edge-Weighted Graphs , 2013, ISMM.

[60]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[61]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[62]  John C Chaput,et al.  Evolutionary optimization of a nonbiological ATP binding protein for improved folding stability. , 2004, Chemistry & biology.