Fast Projection Onto Convex Smooth Constraints

The Euclidean projection onto a convex set is an important problem that arises in numerous constrained optimization tasks. Unfortunately, in many cases, computing projections is computationally demanding. In this work, we focus on projection problems where the constraints are smooth and the number of constraints is significantly smaller than the dimension. The runtime of existing approaches to solving such problems is either cubic in the dimension or polynomial in the inverse of the target accuracy. Conversely, we propose a simple and efficient primal-dual approach, with a runtime that scales only linearly with the dimension, and only logarithmically in the inverse of the target accuracy. We empirically demonstrate its performance, and compare it with standard baselines.

[1]  Jinfeng Yi,et al.  Stochastic Gradient Descent with Only One Projection , 2012, NIPS.

[2]  Pravin M. Vaidya,et al.  A new algorithm for minimizing convex functions over convex sets , 1996, Math. Program..

[3]  Ofer Meshi,et al.  Linear-Memory and Decomposition-Invariant Linearly Convergent Conditional Gradient Algorithm for Structured Polytopes , 2016, NIPS.

[4]  Yin Tat Lee,et al.  A Faster Cutting Plane Method and its Implications for Combinatorial and Convex Optimization , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[5]  Dan Garber,et al.  Faster Projection-free Convex Optimization over the Spectrahedron , 2016, NIPS.

[6]  Martin Jaggi,et al.  Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization , 2013, ICML.

[7]  Michael I. Jordan,et al.  A General Analysis of the Convergence of ADMM , 2015, ICML.

[8]  Laurent Condat.  Fast projection onto the simplex and the l1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pmb {l}_\mathbf {1}$$\end{ , 2015, Mathematical Programming.

[9]  Daniel Pérez Palomar,et al.  Randomized Algorithms for Optimal Solutions of Double-Sided QCQP With Applications in Signal Processing , 2014, IEEE Transactions on Signal Processing.

[10]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[11]  A. Nemirovski,et al.  Interior-point methods for optimization , 2008, Acta Numerica.

[12]  D. J. Newman,et al.  Location of the Maximum on Unimodal Surfaces , 1965, JACM.

[13]  Martin Jaggi,et al.  On the Global Linear Convergence of Frank-Wolfe Optimization Variants , 2015, NIPS.

[14]  Kinjal Basu,et al.  Large-Scale Quadratically Constrained Quadratic Program via Low-Discrepancy Sequences , 2017, NIPS.

[15]  Xiaofeng Cao,et al.  Fully Projection-Free Proximal Stochastic Gradient Method With Optimal Convergence Rates , 2020, IEEE Access.

[16]  Yi Zhou,et al.  Conditional Accelerated Lazy Stochastic Gradient Descent , 2017, ICML.

[17]  Narendra Karmarkar,et al.  A new polynomial-time algorithm for linear programming , 1984, Comb..

[18]  Yin Tat Lee,et al.  An improved cutting plane method for convex optimization, convex-concave games, and its applications , 2020, STOC.

[19]  Éva Tardos,et al.  Fast approximation algorithms for fractional packing and covering problems , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[20]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[21]  Yi Zhou,et al.  Conditional Gradient Sliding for Convex Optimization , 2016, SIAM J. Optim..

[22]  Elad Hazan,et al.  Faster Rates for the Frank-Wolfe Method over Strongly-Convex Sets , 2014, ICML.

[23]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[24]  Elad Hazan,et al.  Playing Non-linear Games with Linear Oracles , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[25]  Stephen P. Boyd,et al.  Metric Selection in Douglas-Rachford Splitting and ADMM , 2014 .

[26]  Yuanzhi Li,et al.  Linear Convergence of a Frank-Wolfe Type Algorithm over Trace-Norm Balls , 2017, NIPS.

[27]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[28]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[29]  Stephen P. Boyd,et al.  Optimal kernel selection in Kernel Fisher discriminant analysis , 2006, ICML.

[30]  David H. Mathews,et al.  Detection of non-coding RNAs on the basis of predicted secondary structure formation free energy change , 2006, BMC Bioinformatics.

[31]  E. Altman.  Constrained Markov Decision Processes , 1999 .

[32]  Zoubin Ghahramani,et al.  Graph Kernels by Spectral Transforms , 2006, Semi-Supervised Learning.

[33]  Murat Dundar,et al.  A fast iterative algorithm for fisher discriminant using heterogeneous kernels , 2004, ICML.

[34]  Andreas Krause,et al.  Projection Free Online Learning over Smooth Sets , 2019, AISTATS.

[35]  N. Z. Shor.  Cut-off method with space extension in convex programming problems , 1977, Cybernetics.

[36]  Mark W. Schmidt,et al.  Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm , 2009, AISTATS.

[37]  Jieping Ye,et al.  Learning the kernel matrix in discriminant analysis via quadratically constrained quadratic programming , 2007, KDD '07.

[38]  B. Wohlberg,et al.  Fast Projection onto the 𝓁 ∞, 1 -Mixed Norm Ball Using Steffensen Root Search. , 2018, ICASSP 2018.

[39]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[40]  Xudong Li,et al.  Fast projection onto the ordered weighted $\ell_1$ norm ball , 2020, 2002.05004.

[41]  Tianbao Yang,et al.  A Richer Theory of Convex Constrained Optimization with Reduced Projections and Improved Rates , 2016, ICML.

[42]  Rekha R. Thomas,et al.  A QCQP Approach to Triangulation , 2012, ECCV.

[43]  Felix J. Herrmann,et al.  Algorithms and software for projections onto intersections of convex and non-convex sets with applications to inverse problems , 2019, ArXiv.

[44]  Zheng Xu,et al.  Adaptive Consensus ADMM for Distributed Optimization , 2017, ICML.

[45]  Sanjeev Arora,et al.  Fast algorithms for approximate semidefinite programming using the multiplicative weights update method , 2005, 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS'05).

[46]  Claude Lemaréchal,et al.  An Algorithm for Minimizing Convex Functions , 1974, IFIP Congress.

[47]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[48]  Aaron Sidford,et al.  Efficiently Solving MDPs with Stochastic Mirror Descent , 2020, ICML.