Inexact model: a framework for optimization and variational inequalities

In this paper we propose a general algorithmic framework for first-order methods in optimization in a broad sense, including minimization problems, saddle-point problems and variational inequalities. This framework allows to obtain many known methods as a special case, the list including accelerated gradient method, composite optimization methods, level-set methods, proximal methods. The idea of the framework is based on constructing an inexact model of the main problem component, i.e. objective function in optimization or operator in variational inequalities. Besides reproducing known results, our framework allows to construct new methods, which we illustrate by constructing a universal method for variational inequalities with composite structure. This method works for smooth and non-smooth problems with optimal complexity without a priori knowledge of the problem smoothness. We also generalize our framework for strongly convex objectives and strongly monotone variational inequalities.

[1]  Darina Dvinskikh,et al.  On Primal and Dual Approaches for Distributed Stochastic Convex Optimization over Networks , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[2]  Yurii Nesterov,et al.  Implementable tensor methods in unconstrained convex optimization , 2019, Mathematical Programming.

[3]  Julien Mairal,et al.  Optimization with First-Order Surrogate Functions , 2013, ICML.

[4]  Sergey Omelchenko,et al.  A Stable Alternative to Sinkhorn's Algorithm for Regularized Optimal Transport , 2017, MOTOR.

[5]  A. V. Gasnikov,et al.  An Adaptive Proximal Method for Variational Inequalities , 2019, Computational Mathematics and Mathematical Physics.

[6]  Zaïd Harchaoui,et al.  A Universal Catalyst for First-Order Optimization , 2015, NIPS.

[7]  Kimon Antonakopoulos,et al.  Adaptive extra-gradient methods for min-max optimization and games , 2020, ICLR.

[8]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[9]  Zaïd Harchaoui,et al.  Conditional gradient algorithms for norm-regularized smooth convex optimization , 2013, Math. Program..

[10]  Salahuddin,et al.  Two-level iterative method for non-stationary mixed variational inequalities , 2017 .

[11]  Jérémie Bigot,et al.  Consistent estimation of a population barycenter in the Wasserstein space , 2013 .

[12]  On the line-search gradient methods for stochastic optimization , 2020 .

[13]  Martin Jaggi,et al.  Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization , 2013, ICML.

[14]  Gasnikov Aleksandr,et al.  Effective Numerical Methods for Huge-Scale Linear Systems with Double-Sparsity and Applications to PageRank , 2015 .

[15]  Alexander Gasnikov,et al.  Gradient Methods for Problems with Inexact Model of the Objective , 2019, MOTOR.

[16]  Yurii Nesterov,et al.  Cubic regularization of Newton method and its global performance , 2006, Math. Program..

[17]  Uriel G. Rothblum,et al.  Accuracy Certificates for Computational Problems with Convex Structure , 2010, Math. Oper. Res..

[18]  Yangyang Xu,et al.  Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems , 2018, Math. Program..

[19]  Arkadi Nemirovski,et al.  Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems , 2004, SIAM J. Optim..

[20]  Jason Altschuler,et al.  Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration , 2017, NIPS.

[21]  Angelia Nedic,et al.  Distributed Computation of Wasserstein Barycenters Over Networks , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[22]  Mohammad S. Alkousa,et al.  On Some Methods for Strongly Convex Optimization Problems with One Functional Constraint , 2019, MOTOR.

[23]  Yu Malitsky,et al.  Proximal extrapolated gradient methods for variational inequalities , 2016, Optim. Methods Softw..

[24]  Alexander Gasnikov,et al.  Efficient numerical methods to solve sparse linear equations with application to PageRank , 2015, Optim. Methods Softw..

[25]  Marc Teboulle,et al.  A Descent Lemma Beyond Lipschitz Gradient Continuity: First-Order Methods Revisited and Applications , 2017, Math. Oper. Res..

[26]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[27]  Thibaut Le Gouic,et al.  Existence and consistency of Wasserstein barycenters , 2015, Probability Theory and Related Fields.

[28]  Guanghui Lan,et al.  An optimal method for stochastic composite optimization , 2011, Mathematical Programming.

[29]  Alexander Gasnikov,et al.  Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm , 2018, ICML.

[30]  P. Dvurechensky,et al.  Generalized Mirror Prox: Solving Variational Inequalities with Monotone Operator, Inexact Oracle, and Unknown H\"older Parameters , 2018 .

[31]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[32]  Gleb Gusev,et al.  Learning Supervised PageRank with Gradient-Based and Gradient-Free Optimization Methods , 2016, NIPS.

[33]  Y. Nesterov,et al.  Accelerated Primal-Dual Gradient Descent with Linesearch for Convex, Nonconvex, and Nonsmooth Optimization Problems , 2019, Doklady Mathematics.

[34]  A. V. Gasnikov,et al.  Stochastic intermediate gradient method for convex optimization problems , 2016 .

[35]  Kevin Tian,et al.  Relative Lipschitzness in Extragradient Methods and a Direct Recipe for Acceleration , 2020, ITCS.

[36]  Marco Cuturi,et al.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport , 2013, NIPS.

[37]  Yin Tat Lee,et al.  Near Optimal Methods for Minimizing Convex Functions with Lipschitz $p$-th Derivatives , 2019, COLT.

[38]  Michael I. Jordan,et al.  On Efficient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithms , 2019, ICML.

[39]  Yurii Nesterov,et al.  Universal gradient methods for convex optimization problems , 2015, Math. Program..

[40]  A. V. Gasnikov,et al.  Fast Gradient Descent for Convex Minimization Problems with an Oracle Producing a (δ, L)-Model of Function at the Requested Point , 2019, Computational Mathematics and Mathematical Physics.

[41]  Alexander Gasnikov,et al.  Randomized Similar Triangles Method: A Unifying Framework for Accelerated Randomized Optimization Methods (Coordinate Descent, Directional Search, Derivative-Free Method) , 2017, ArXiv.

[42]  Fedor S. Stonyakin.  Some adaptive proximal method for a special class of abstract variational inequalities , 2019 .

[43]  Yurii Nesterov,et al.  Primal-dual subgradient methods for convex problems , 2005, Math. Program..

[44]  Aaron Defazio,et al.  A Simple Practical Accelerated Method for Finite Sums , 2016, NIPS.

[45]  Laurent Massoulié,et al.  Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks , 2017, ICML.

[46]  J. Lorenz,et al.  On the scaling of multidimensional matrices , 1989 .

[47]  Mohamed-Jalal Fadili,et al.  Non-smooth Non-convex Bregman Minimization: Unification and New Algorithms , 2017, Journal of Optimization Theory and Applications.

[48]  Darina Dvinskikh,et al.  Decentralize and Randomize: Faster Algorithm for Wasserstein Barycenters , 2018, NeurIPS.

[49]  Alexander Gasnikov,et al.  Fast gradient descent method for convex optimization problems with an oracle that generates a $(\delta,L)$-model of a function in a requested point , 2017, 1711.02747.

[50]  Yurii Nesterov,et al.  Complexity bounds for primal-dual methods minimizing the model of objective function , 2017, Mathematical Programming.

[51]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[52]  F. Stonyakin.  Some adaptive proximal method for a special class of variational inequalities , 2019, 1901.02800.

[53]  G. Mastroeni.  On Auxiliary Principle for Equilibrium Problems , 2003 .

[54]  Yurii Nesterov,et al.  Relatively Smooth Convex Optimization by First-Order Methods, and Applications , 2016, SIAM J. Optim..

[55]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[56]  Yurii Nesterov.  Soft clustering by convex electoral model , 2020, Soft Comput..

[57]  Yura Malitsky,et al.  Golden ratio algorithms for variational inequalities , 2018, Mathematical Programming.

[58]  V. Spokoiny,et al.  Construction of Non-asymptotic Confidence Sets in 2-Wasserstein Space , 2017, 1703.03658.

[59]  Kimon Antonakopoulos,et al.  An adaptive Mirror-Prox method for variational inequalities with singular operators , 2019, NeurIPS.

[60]  A. V. Gasnikov,et al.  Universal Method for Stochastic Composite Optimization Problems , 2018 .

[61]  Aaron Sidford,et al.  Towards Optimal Running Times for Optimal Transport , 2018, ArXiv.

[62]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[63]  P. Dvurechensky,et al.  Dual approaches to the minimization of strongly convex functionals with a simple structure under affine constraints , 2017 .

[64]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[65]  Darina Dvinskikh,et al.  On the Complexity of Approximating Wasserstein Barycenter , 2019, ArXiv.

[66]  Alexander Gasnikov,et al.  Stochastic Intermediate Gradient Method for Convex Problems with Stochastic Inexact Oracle , 2016, Journal of Optimization Theory and Applications.

[67]  Cristobal Guzman,et al.  On lower complexity bounds for large-scale smooth convex optimization , 2013, J. Complex..

[68]  Nesterov Yurii,et al.  Gradient methods with memory , 2019 .

[69]  Guanghui Lan,et al.  Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization , 2013, Mathematical Programming.

[70]  Anton Rodomanov,et al.  Primal-Dual Method for Searching Equilibrium in Hierarchical Congestion Population Games , 2016, DOOR.

[71]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[72]  E. Barrio,et al.  A statistical analysis of a deformation model with Wasserstein barycenters : estimation procedure and goodness of fit test , 2015, 1508.06465.

[73]  Hongyuan Zha,et al.  A Fast Proximal Point Method for Wasserstein Distance , 2018, ArXiv.

[74]  Marc Teboulle,et al.  Convergence Analysis of a Proximal-Like Minimization Algorithm Using Bregman Functions , 1993, SIAM J. Optim..

[75]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[76]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[77]  Alexey Chernov,et al.  Fast Primal-Dual Gradient Method for Strongly Convex Minimization Problems with Linear Constraints , 2016, DOOR.

[78]  Y. Nesterov,et al.  First-order methods with inexact oracle: the strongly convex case , 2013 .

[79]  Alexandre d'Aspremont,et al.  Optimal Complexity and Certification of Bregman First-Order Methods , 2021, Mathematical Programming.

[80]  Alexander Gasnikov,et al.  Universal method with inexact oracle and its applications for searching equillibriums in multistage transport problems , 2015 .

[81]  L. Kantorovich.  On the Translocation of Masses , 2006 .

[82]  Alexander Gasnikov,et al.  Generalized Mirror Prox for Monotone Variational Inequalities: Universality and Inexact Oracle , 2018, 1806.05140.

[83]  A. Nemirovskii,et al.  Optimal methods of smooth convex minimization , 1986 .

[84]  A. Gasnikov.  Universal gradient descent , 2017, 1711.00394.

[85]  P. Dvurechensky,et al.  Universal intermediate gradient method for convex problems with inexact oracle , 2017, Optim. Methods Softw..

[86]  Leonidas J. Guibas,et al.  Wasserstein Propagation for Semi-Supervised Learning , 2014, ICML.

[87]  Dmitriy Drusvyatskiy,et al.  Nonsmooth optimization using Taylor-like models: error bounds, convergence, and termination criteria , 2016, Mathematical Programming.

[88]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[89]  Alfredo N. Iusem,et al.  Variance-Based Extragradient Methods with Line Search for Stochastic Variational Inequalities , 2019, SIAM J. Optim..

[90]  HarchaouiZaid,et al.  Conditional gradient algorithms for norm-regularized smooth convex optimization , 2015 .

[91]  Phan Quoc Khanh,et al.  SOME ALGORITHMS FOR SOLVING MIXED VARIATIONAL INEQUALITIES , 2006 .

[92]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[93]  Amir Beck,et al.  On the Convergence of Alternating Minimization for Convex Programming with Applications to Iteratively Reweighted Least Squares and Decomposition Schemes , 2015, SIAM J. Optim..

[94]  Yi Zhou,et al.  Random gradient extrapolation for distributed and stochastic optimization , 2017, SIAM J. Optim..

[95]  Yurii Nesterov,et al.  Lectures on Convex Optimization , 2018 .