Alternating Projections and Douglas-Rachford for Sparse Affine Feasibility

The problem of finding a vector with the fewest nonzero elements that satisfies an underdetermined system of linear equations is an NP-complete problem that is typically solved numerically via convex heuristics or nicely-behaved nonconvex relaxations. In this work we consider elementary methods based on projections for solving a sparse feasibility problem without employing convex heuristics. It has been shown recently that, locally, the fundamental method of alternating projections must converge linearly to a solution to the sparse feasibility problem with an affine constraint. In this paper we apply different analytical tools that allow us to show global linear convergence of alternating projections under familiar constraint qualifications. These analytical tools can also be applied to other algorithms. This is demonstrated with the prominent Douglas-Rachford algorithm where we establish local linear convergence of this method applied to the sparse affine feasibility problem.

[1]  Hung M. Phan,et al.  Linear convergence of the Douglas–Rachford method for two closed sets , 2014, 1401.6509.

[2]  Volkan Cevher,et al.  Matrix Recipes for Hard Thresholding Methods , 2012, Journal of Mathematical Imaging and Vision.

[3]  D. Russell Luke,et al.  Nonconvex Notions of Regularity and Convergence of Fundamental Algorithms for Feasibility Problems , 2012, SIAM J. Optim..

[4]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[5]  Heinz H. Bauschke,et al.  Restricted Normal Cones and the Method of Alternating Projections: Applications , 2012, 1205.0318.

[6]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[7]  Hai Yen Le.  Generalized subdifferentials of the rank function , 2013, Optim. Lett..

[8]  Boris Polyak,et al.  B.S. Mordukhovich. Variational Analysis and Generalized Differentiation. I. Basic Theory, II. Applications , 2009 .

[9]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[10]  Yonina C. Eldar,et al.  Phase Retrieval via Matrix Completion , 2011, SIAM Rev..

[11]  D. Russell Luke.  Prox-Regularity of Rank Constraint Sets and Implications for Algorithms , 2012, Journal of Mathematical Imaging and Vision.

[12]  Heinz H. Bauschke,et al.  On the convergence of von Neumann's alternating projection algorithm for two sets , 1993 .

[13]  E. H. Zarantonello.  Projections on Convex Sets in Hilbert Space and Spectral Theory: Part I. Projections on Convex Sets: Part II. Spectral Theory , 1971 .

[14]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[15]  Michel Théra,et al.  Metric Inequality, Subdifferential Calculus and Applications , 2001 .

[16]  M. Lai,et al.  An Unconstrained $\ell_q$ Minimization with $0q\leq1$ for Sparse Solution of Underdetermined Linear Systems , 2011 .

[17]  E. Polak.  Method of Successive Projections for Finding a Common Point of Sets in Metric Spaces , 1990 .

[18]  Heinz H. Bauschke,et al.  Finding best approximation pairs relative to two closed convex sets in Hilbert spaces , 2004, J. Approx. Theory.

[19]  Guy Pierra,et al.  Decomposition through formalization in a product space , 1984, Math. Program..

[20]  Jonathan M. Borwein,et al.  Entropic Regularization of the ℓ 0 Function , 2011, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[21]  Heinz H. Bauschke,et al.  The rate of linear convergence of the Douglas-Rachford algorithm for subspaces is the cosine of the Friedrichs angle , 2013, J. Approx. Theory.

[22]  Laurent Demanet,et al.  Eventual linear convergence of the Douglas-Rachford iteration for basis pursuit , 2013, Math. Comput..

[23]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[24]  Jean-Baptiste Hiriart-Urruty,et al.  When only global optimization matters , 2013, J. Glob. Optim..

[25]  Guy Pierra,et al.  Eclatement de Contraintes en Parallèle pour la Minimisation d'une Forme Quadratique , 1975, Optimization Techniques.

[26]  Ming-Jun Lai,et al.  An Unconstrained ℓq Minimization with 0 , 2011, SIAM J. Optim..

[27]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[28]  A. Ioffe.  Metric regularity and subdifferential calculus , 2000 .

[29]  Mike E. Davies,et al.  Normalized Iterative Hard Thresholding: Guaranteed Stability and Performance , 2010, IEEE Journal of Selected Topics in Signal Processing.

[30]  Bangti Jin,et al.  Heuristic Parameter-Choice Rules for Convex Variational Regularization Based on Error Estimates , 2010, SIAM J. Numer. Anal..

[31]  Adrian S. Lewis,et al.  Local Linear Convergence for Alternating and Averaged Nonconvex Projections , 2009, Found. Comput. Math..

[32]  Marc Teboulle,et al.  A Linearly Convergent Algorithm for Solving a Class of Nonconvex/Affine Feasibility Problems , 2011, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[33]  Marc E. Pfetsch,et al.  The Computational Complexity of the Restricted Isometry Property, the Nullspace Property, and Related Concepts in Compressed Sensing , 2012, IEEE Transactions on Information Theory.

[34]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[35]  Heinz H. Bauschke,et al.  A strongly convergent reflection method for finding the projection onto the intersection of two closed convex sets in a Hilbert space , 2006, J. Approx. Theory.

[36]  Heinz H. Bauschke,et al.  Restricted Normal Cones and the Method of Alternating Projections: Theory , 2012 .

[37]  Heinz H. Bauschke,et al.  Restricted Normal Cones and Sparsity Optimization with Affine Constraints , 2012, Found. Comput. Math..

[38]  A. Kruger.  About Regularity of Collections of Sets , 2006 .

[39]  D. Russell Luke,et al.  Finding Best Approximation Pairs Relative to a Convex and Prox-Regular Set in a Hilbert Space , 2008, SIAM J. Optim..

[40]  Heinz H. Bauschke,et al.  On the local convergence of the Douglas–Rachford algorithm , 2014, 1401.6188.