Random Gradient-Free Minimization of Convex Functions

In this paper, we prove new complexity bounds for methods of convex optimization based only on computation of the function value. The search directions of our schemes are normally distributed random Gaussian vectors. It appears that such methods usually need at most n times more iterations than the standard gradient methods, where n is the dimension of the space of variables. This conclusion is true for both nonsmooth and smooth problems. For the latter class, we present also an accelerated scheme with the expected rate of convergence $$O\Big ({n^2 \over k^2}\Big )$$O(n2k2), where k is the iteration counter. For stochastic optimization, we propose a zero-order scheme and justify its expected rate of convergence $$O\Big ({n \over k^{1/2}}\Big )$$O(nk1/2). We give also some bounds for the rate of convergence of the random gradient-free methods to stationary points of nonconvex functions, for both smooth and nonsmooth cases. Our theoretical results are supported by preliminary computational experiments.

[1]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[2]  References , 1971 .

[3]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[4]  F. Clarke Optimization And Nonsmooth Analysis , 1983 .

[5]  Y. Pang Expected number of steps of a random optimization method , 1985 .

[6]  M. Sarma On the convergence of the Baba and Dorea random optimization methods , 1990 .

[7]  V. Protasov Algorithms for approximate calculation of the minimum of a convex function from its values , 1996 .

[8]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[9]  Christian Gourieroux,et al.  Simulation-based econometric methods , 1996 .

[10]  Jeffrey C. Lagarias,et al.  Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions , 1998, SIAM J. Optim..

[11]  J. Huriot,et al.  Economics of Cities , 2000 .

[12]  James Renegar,et al.  A mathematical view of interior-point methods in convex optimization , 2001, MPS-SIAM series on optimization.

[13]  Daniel Bienstock,et al.  Potential Function Methods for Approximately Solving Linear Programming Problems: Theory and Practice , 2002 .

[14]  Santosh S. Vempala,et al.  Solving convex programs by random walks , 2004, JACM.

[15]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[16]  Yurii Nesterov,et al.  Lexicographic differentiation of nonsmooth functions , 2005, Math. Program..

[17]  Adam Tauman Kalai,et al.  Online convex optimization in the bandit setting: gradient descent without a gradient , 2004, SODA '05.

[18]  J. Gabszewicz La différenciation des produits , 2006 .

[19]  Laurence A. Wolsey,et al.  Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, 4th International Conference, CPAIOR 2007, Brussels, Belgium, May 23-26, 2007, Proceedings , 2007, CPAIOR.

[20]  Nicolas Gillis,et al.  Nonnegative Factorization and The Maximum Edge Biclique Problem , 2008, 0810.4225.

[21]  Eli Upfal,et al.  Multi-Armed Bandits in Metric Spaces ∗ , 2008 .

[22]  Jacques-François Thisse,et al.  Economic Geography: The Integration of Regions and Nations , 2008 .

[23]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[24]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[25]  P. Pestieau,et al.  Fertility, Human Capital Accumulation, and the Pension System , 2009, SSRN Electronic Journal.

[26]  G. Oggioni,et al.  Generalized Nash Equilibrium and market coupling in the European power system , 2010 .

[27]  Victor Ginsburgh,et al.  Success: talent, intelligence or beauty? , 2010 .

[28]  Lin Xiao,et al.  Optimal Algorithms for Online Convex Optimization with Multi-Point Bandit Feedback. , 2010, COLT 2010.

[29]  E. Ramaekers Fair allocation of indivisible goods among two agents , 2010 .

[30]  Per J. Agrell,et al.  Dynamic joint investments in supply chains under information asymmetry , 2010 .

[31]  Thierry Bréchet,et al.  Adaptation and Mitigation in Long-term Climate Policy , 2010, Environmental and Resource Economics.

[32]  J. Gabszewicz,et al.  Product innovation and market acquisition of firms , 2010 .

[33]  Thierry Bréchet,et al.  Property rights with biological spillovers: when Hardin meets Meade , 2010 .

[34]  J. Hindriks,et al.  School tracking, social segregation and educational opportunity: evidence from Belgium , 2010 .

[35]  T. Baudin The optimal trade-off between quality and quantity with uncertain child survival , 2010 .

[36]  Jean-Pierre Florens,et al.  Nonparametric frontier estimation from noisy data , 2010 .

[37]  J. Hindriks,et al.  School autonomy and educational performance: within-country evidence , 2010 .

[38]  John L. Nazareth,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[39]  G. Oggioni,et al.  Market coupling and the organization of counter-trading: separating energy and transmission again? , 2010 .

[40]  Paul Belleflamme,et al.  Industrial Organization: Markets and Strategies , 2010 .

[41]  F. Schroyen,et al.  Optimal pricing and capacity choice for a public service under risk of interruption , 2011 .

[42]  Sham M. Kakade,et al.  Stochastic Convex Optimization with Bandit Feedback , 2011, SIAM J. Optim..

[43]  Laurence A. Wolsey,et al.  Mixing Sets Linked by Bidirected Paths , 2011, SIAM J. Optim..

[44]  Thierry Bréchet,et al.  The Benefits of Cooperation Under Uncertainty: the Case of Climate Change , 2011, Environmental Modeling & Assessment.

[45]  Nicolas Gillis,et al.  Low-Rank Matrix Approximation with Weights or Missing Data Is NP-Hard , 2010, SIAM J. Matrix Anal. Appl..

[46]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[47]  Nicolas Gillis,et al.  On the Geometric Interpretation of the Nonnegative Rank , 2010, 1009.0880.

[48]  Martin J. Wainwright,et al.  Finite Sample Convergence Rates of Zero-Order Stochastic Optimization Methods , 2012, NIPS.

[49]  Jeffrey C. Lagarias,et al.  Convergence of the Restricted Nelder-Mead Algorithm in Two Dimensions , 2011, SIAM J. Optim..

[50]  Axel Gautier,et al.  Competitively neutral universal service obligations , 2012, Inf. Econ. Policy.

[51]  J. Johannes,et al.  Iterative regularization in nonparametric instrumental regression , 2013 .

[52]  A. Mauleon,et al.  Bargaining and Delay in Patent Licensing , 2013 .

[53]  Shlomo Weber,et al.  Stability and fairness in models with a multiple membership , 2011, International Journal of Game Theory.

[54]  Thierry Bréchet,et al.  Tradable pollution permits in dynamic general equilibrium: can optimality and acceptability be reconciled? , 2013 .

[55]  J. Rombouts,et al.  Option Pricing with Asymmetric Heteroskedastic Normal Mixture Models , 2010 .

[56]  Per J. Agrell,et al.  A Theory of Soft Capture , 2012 .