Linear convergence of the randomized sparse Kaczmarz method

The randomized version of the Kaczmarz method for the solution of consistent linear systems is known to converge linearly in expectation. And even in the possibly inconsistent case, when only noisy data is given, the iterates are expected to reach an error threshold in the order of the noise-level with the same rate as in the noiseless case. In this work we show that the same also holds for the iterates of the recently proposed randomized sparse Kaczmarz method for recovery of sparse solutions. Furthermore we consider the more general setting of convex feasibility problems and their solution by the method of randomized Bregman projections. This is motivated by the observation that, similarly to the Kaczmarz method, the Sparse Kaczmarz method can also be interpreted as an iterative Bregman projection method to solve a convex feasibility problem. We obtain expected sublinear rates for Bregman projections with respect to a general strongly convex function. Moreover, even linear rates are expected for Bregman projections with respect to smooth or piecewise linear-quadratic functions, and also the regularized nuclear norm, which is used in the area of low rank matrix problems.

[1]  Deanna Needell,et al.  Block Kaczmarz Method with Inequalities , 2014, Journal of Mathematical Imaging and Vision.

[2]  D. Needell.  Randomized Kaczmarz solver for noisy linear systems , 2009, 0902.0958.

[3]  S. Petra,et al.  Randomized Sparse Block Kaczmarz as Randomized Dual Block-Coordinate Descent , 2015 .

[4]  Wotao Yin,et al.  Analysis and Generalizations of the Linearized Bregman Method , 2010, SIAM J. Imaging Sci..

[5]  Hassan Mansour,et al.  A sparse randomized Kaczmarz algorithm , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[6]  L. Bregman.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming , 1967 .

[7]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[8]  Xuemei Chen,et al.  Almost Sure Convergence of the Kaczmarz Algorithm with Random Measurements , 2012 .

[9]  Yair Censor,et al.  A multiprojection algorithm using Bregman projections in a product space , 1994, Numerical Algorithms.

[10]  Qingzhi Yang,et al.  Several solution methods for the split feasibility problem , 2005 .

[11]  Heinz H. Bauschke,et al.  Strong conical hull intersection property, bounded linear regularity, Jameson’s property (G), and error bounds in convex optimization , 1999, Math. Program..

[12]  M. Brand,et al.  Fast low-rank modifications of the thin singular value decomposition , 2006 .

[13]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[14]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[15]  Hui Zhang,et al.  Restricted strong convexity and its applications to convergence analysis of gradient-type methods in convex optimization , 2015, Optim. Lett..

[16]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[17]  Frank Schöpfer,et al.  Linear Convergence of Descent Methods for the Unconstrained Minimization of Restricted Strongly Convex Functions , 2016, SIAM J. Optim..

[18]  C. Byrne,et al.  A unified treatment of some iterative algorithms in signal processing and image reconstruction , 2003 .

[19]  Marcus A. Magnor,et al.  A sparse Kaczmarz solver and a linearized Bregman method for online compressed sensing , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[20]  F. Schöpfer,et al.  An iterative regularization method for the solution of the split feasibility problem in Banach spaces , 2008 .

[21]  Wotao Yin,et al.  Augmented 퓁1 and Nuclear-Norm Models with a Globally Linearly Convergent Algorithm , 2012, SIAM J. Imaging Sci..

[22]  Hein Hundal,et al.  The Rate of Convergence for the Method of Alternating Projections, II , 1997 .

[23]  Martin Burger,et al.  Bregman Distances in Inverse Problems and Partial Differential Equations , 2015, 1505.05191.

[24]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[25]  Dimitri P. Bertsekas,et al.  Stochastic First-Order Methods with Random Constraint Projection , 2016, SIAM J. Optim..

[26]  D. Butnariu,et al.  Convergence of Bregman Projection Methods for Solving Consistent Convex Feasibility Problems in Reflexive Banach Spaces , 1997 .

[27]  Deanna Needell,et al.  Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm , 2013, Mathematical Programming.

[28]  C. Byrne,et al.  Iterative oblique projection onto convex sets and the split feasibility problem , 2002 .

[29]  Y. Censor,et al.  The multiple-sets split feasibility problem and its applications for inverse problems , 2005 .

[30]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[31]  Frank Schöpfer.  Exact Regularization of Polyhedral Norms , 2012, SIAM J. Optim..

[32]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[33]  Yair Censor,et al.  Proximity Function Minimization Using Multiple Bregman Projections, with Applications to Split Feasibility and Kullback–Leibler Distance Minimization , 2001, Ann. Oper. Res..

[34]  Hui Zhang,et al.  Strongly Convex Programming for Exact Matrix Completion and Robust Principal Component Analysis , 2011, ArXiv.

[35]  Jian-Feng Cai,et al.  Convergence of the linearized Bregman iteration for ℓ1-norm minimization , 2009, Math. Comput..

[36]  Heinz H. Bauschke,et al.  Legendre functions and the method of random Bregman projections , 1997 .

[37]  Adrian S. Lewis,et al.  Randomized Methods for Linear Constraints: Convergence Rates and Conditioning , 2008, Math. Oper. Res..

[38]  Yue M. Lu,et al.  Randomized Kaczmarz algorithms: Exact MSE analysis and optimal sampling probabilities , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[39]  S. M. Robinson.  Some continuity properties of polyhedral multifunctions , 1981 .

[40]  R. Tyrrell Rockafellar,et al.  Variational Analysis , 1998, Grundlehren der mathematischen Wissenschaften.

[41]  Per Christian Hansen,et al.  AIR Tools - A MATLAB package of algebraic iterative reconstruction methods , 2012, J. Comput. Appl. Math..

[42]  Deanna Needell,et al.  Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method , 2012, ArXiv.

[43]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[44]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[45]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[46]  Dirk A. Lorenz,et al.  The Linearized Bregman Method via Split Feasibility Problems: Analysis and Generalizations , 2013, SIAM J. Imaging Sci..

[47]  Heinz H. Bauschke,et al.  Bregman Monotone Optimization Algorithms , 2003, SIAM J. Control. Optim..

[48]  A. Hoffman.  On approximate solutions of systems of linear inequalities , 1952 .

[49]  Hassan Mansour,et al.  A fast randomized Kaczmarz algorithm for sparse solutions of consistent linear systems , 2013, ArXiv.