Stochastic Three Points Method for Unconstrained Smooth Minimization

In this paper we consider the unconstrained minimization problem of a smooth function in ${\mathbb{R}}^n$ in a setting where only function evaluations are possible. We design a novel randomized derivative-free algorithm --- the stochastic three points (STP) method --- and analyze its iteration complexity. At each iteration, STP generates a random search direction according to a certain fixed probability law. Our assumptions on this law are very mild: roughly speaking, all laws which do not concentrate all measure on any halfspace passing through the origin will work. For instance, we allow for the uniform distribution on the sphere and also distributions that concentrate all measure on a positive spanning set. Given a current iterate $x$, STP compares the objective function at three points: $x$, $x+\alpha s$ and $x-\alpha s$, where $\alpha>0$ is a stepsize parameter and $s$ is the random search direction. The best of these three points is the next iterate. We analyze the method STP under several stepsize selection schemes (fixed, decreasing, estimated through finite differences, etc). We study non-convex, convex and strongly convex cases.

[1]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[2]  J. M. Martínez,et al.  A derivative-free nonmonotone line-search technique for unconstrained optimization , 2008 .

[3]  Luís Nunes Vicente,et al.  Worst case complexity of direct search , 2013, EURO J. Comput. Optim..

[4]  Yurii Nesterov,et al.  Random Gradient-Free Minimization of Convex Functions , 2015, Foundations of Computational Mathematics.

[5]  Serge Gratton,et al.  Direct Search Based on Probabilistic Descent , 2015, SIAM J. Optim..

[6]  Tamara G. Kolda,et al.  Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods , 2003, SIAM Rev..

[7]  N. Baba.  Convergence of a random optimization method for constrained optimization problems , 1981 .

[8]  O. Pironneau,et al.  Applied Shape Optimization for Fluids , 2001 .

[9]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[10]  V. G. Karmanov.  On Convergence of a Random Search Method in Convex Minimization Problems , 1975 .

[11]  V. G. Karmanov.  Convergence estimates for iterative minimization methods , 1974 .

[12]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[13]  G. Allaire,et al.  Shape optimization by the homogenization method , 1997 .

[14]  Luís N. Vicente,et al.  Worst case complexity of direct search under convexity , 2016, Math. Program..

[15]  Y. Pang.  Expected number of steps of a random optimization method , 1985 .

[16]  Raino A. E. Mäkinen,et al.  Introduction to shape optimization - theory, approximation, and computation , 2003, Advances in design and control.

[17]  Jan Sokolowski,et al.  Introduction to shape optimization , 1992 .

[18]  John L. Nazareth,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[19]  Peter Richt'arik,et al.  Simple Complexity Analysis of Simplified Direct Search , 2014, 1410.0390.

[20]  Eduard A. Gorbunov,et al.  An Accelerated Method for Derivative-Free Smooth Stochastic Convex Optimization , 2018, SIAM J. Optim..

[21]  Christian L. Müller,et al.  Optimization of Convex Functions with Random Pursuit , 2011, SIAM J. Optim..

[22]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[23]  M. Sarma.  On the convergence of the Baba and Dorea random optimization methods , 1990 .