Optimal and Adaptive Monteiro-Svaiter Acceleration

We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration. Consequently, for any p ≥ 2 we improve the complexity of convex optimization with Lipschitz p th derivative by a logarithmic factor, matching a lower bound. We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second- or first-order method via exact linear system solution or MinRes, respectively. On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton’s method; simply iterating our first-order adaptive subproblem solver performs comparably to L-BFGS

[1]  A. Gasnikov,et al.  The First Optimal Acceleration of High-Order Methods in Smooth Convex Optimization , 2022, NeurIPS.

[2]  Y. Carmon,et al.  Distributionally Robust Optimization via Ball Oracle Acceleration , 2022, NeurIPS.

[3]  Aryan Mokhtari,et al.  Generalized Optimistic Methods for Convex-Concave Saddle Point Problems , 2022, ArXiv.

[4]  Variants of the A-HPE and large-step A-HPE algorithms for strongly convex problems with applications to accelerated high-order tensor methods , 2021, 2102.02045.

[5]  Kevin A. Lai,et al.  Higher-order methods for convex-concave min-max optimization and monotone variational inequalities , 2020, SIAM J. Optim..

[6]  Michael I. Jordan,et al.  Perseus: A Simple High-Order Regularization Method for Variational Inequalities , 2022, ArXiv.

[7]  Yurii Nesterov,et al.  Gradient regularization of Newton method with Bregman distances , 2021, Math. Program..

[8]  Konstantin Mishchenko.  Regularized Newton Method with Global O(1/k2) Convergence , 2021, ArXiv.

[9]  Ernest K. Ryu,et al.  A Geometric Structure of Acceleration and Its Role in Making Gradients Small Fast , 2021, NeurIPS.

[10]  Yair Carmon,et al.  Stochastic Bias-Reduced Gradient Methods , 2021, NeurIPS.

[11]  Yair Carmon,et al.  Thinking Inside the Ball: Near-Optimal Minimization of the Maximal Loss , 2021, COLT.

[12]  Yurii Nesterov,et al.  Superfast Second-Order Methods for Unconstrained Convex Optimization , 2020, Journal of Optimization Theory and Applications.

[13]  Yurii Nesterov,et al.  Implementable tensor methods in unconstrained convex optimization , 2019, Mathematical Programming.

[14]  Brian Bullins,et al.  Highly smooth minimization of non-smooth problems , 2020, COLT.

[15]  Aaron Sidford,et al.  Acceleration with a Ball Optimization Oracle , 2020, NeurIPS.

[16]  Yurii Nesterov,et al.  Inexact Tensor Methods with Dynamic Accuracies , 2020, ICML.

[17]  D. Kamzolov.  Near-Optimal Hyperfast Second-Order Method for Convex Optimization , 2020, 2002.09050.

[18]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[19]  Y. Nesterov,et al.  Tensor Methods for Minimizing Convex Functions with Hölder Continuous Higher-Order Derivatives , 2019, SIAM J. Optim..

[20]  Bo Jiang,et al.  A Unified Adaptive Tensor Approximation Scheme to Accelerate Composite Convex Optimization , 2020, SIAM J. Optim..

[21]  Yi Ma,et al.  Towards Unified Acceleration of High-Order Algorithms under Hölder Continuity and Uniform Convexity , 2019, ArXiv.

[22]  Yin Tat Lee,et al.  Complexity of Highly Parallel Non-Smooth Convex Optimization , 2019, NeurIPS.

[23]  Eduard A. Gorbunov,et al.  Optimal Tensor Methods in Smooth Convex and Uniformly ConvexOptimization , 2019, COLT.

[24]  Yin Tat Lee,et al.  Near-optimal method for highly smooth convex optimization , 2018, COLT.

[25]  Shuzhong Zhang,et al.  An Optimal High-Order Tensor Method for Convex Optimization , 2019, COLT.

[26]  Ohad Shamir,et al.  Oracle complexity of second-order methods for smooth convex optimization , 2017, Mathematical Programming.

[27]  Yurii Nesterov,et al.  Lectures on Convex Optimization , 2018 .

[28]  Renato D. C. Monteiro,et al.  An Accelerated Hybrid Proximal Extragradient Method for Convex Optimization and Its Implications to Second-Order Methods , 2013, SIAM J. Optim..

[29]  Nicholas I. M. Gould,et al.  Updating the regularization parameter in the adaptive cubic regularization algorithm , 2012, Comput. Optim. Appl..

[30]  Renato D. C. Monteiro,et al.  Iteration-Complexity of a Newton Proximal Extragradient Method for Monotone Variational Inequalities and Inclusion Problems , 2012, SIAM J. Optim..

[31]  Michael A. Saunders,et al.  CG Versus MINRES: An Empirical Comparison , 2012 .

[32]  Nicholas I. M. Gould,et al.  Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity , 2011, Math. Program..

[33]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[34]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[35]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[36]  Saverio Salzo,et al.  Inexact and accelerated proximal point algorithms , 2011 .

[37]  M. Baes.  Estimate sequence methods: extensions and approximations , 2009 .

[38]  Yurii Nesterov,et al.  Accelerating the cubic regularization of Newton’s method on convex problems , 2005, Math. Program..

[39]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[40]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[41]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[42]  Osman Güler,et al.  New Proximal Point Algorithms for Convex Minimization , 1992, SIAM J. Optim..

[43]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[44]  E. Stiefel,et al.  Relaxationsmethoden bester Strategie zur Lösung linearer Gleichungssysteme , 1955 .