A Doubly Stochastic Gauss-Seidel Algorithm for Solving Linear Equations and Certain Convex Minimization Problems

Consider the classical problem of solving a general linear system of equations $Ax=b$. It is well known that the (successively over relaxed) Gauss-Seidel scheme and many of its variants may not converge when $A$ is neither diagonally dominant nor symmetric positive definite. Can we have a linearly convergent G-S type algorithm that works for any $A$? In this paper we answer this question affirmatively by proposing a doubly stochastic G-S algorithm that is provably linearly convergent (in the mean square error sense) for any feasible linear system of equations. The key in the algorithm design is to introduce a nonuniform double stochastic scheme for picking the equation and the variable in each update step as well as a stepsize rule. These techniques also generalize to certain iterative alternating projection algorithms for solving the linear feasibility problem $A x\le b$ with an arbitrary $A$, as well as certain high-dimensional convex minimization problems. Our results demonstrate that a carefully designed randomization scheme can make an otherwise divergent G-S algorithm converge.

[1]  Samuel Awoniyi,et al.  A new iterative method for solving non-square systems of linear equations , 2017, J. Comput. Appl. Math..

[2]  Jesús A. De Loera,et al.  A Sampling Kaczmarz-Motzkin Algorithm for Linear Feasibility , 2016, SIAM J. Sci. Comput..

[3]  Weiqi Zhou,et al.  Random reordering in SOR-type methods , 2017, Numerische Mathematik.

[4]  Deanna Needell,et al.  Rows vs. Columns: Randomized Kaczmarz or Gauss-Seidel for Ridge Regression , 2015, 1507.05844.

[5]  Peter Richtárik,et al.  Semi-stochastic coordinate descent , 2014, Optim. Methods Softw..

[6]  Stephen J. Wright,et al.  An accelerated randomized Kaczmarz algorithm , 2013, Math. Comput..

[7]  Karin Schwab,et al.  Best Approximation In Inner Product Spaces , 2016 .

[8]  Peter Richtárik,et al.  Stochastic Dual Ascent for Solving Linear Systems , 2015, ArXiv.

[9]  P. Oswald,et al.  Convergence analysis for Kaczmarz-type methods in a Hilbert space framework , 2015 .

[10]  Deanna Needell,et al.  Convergence Properties of the Randomized Extended Gauss-Seidel and Kaczmarz Methods , 2015, SIAM J. Matrix Anal. Appl..

[11]  Stephen J. Wright,et al.  An asynchronous parallel stochastic coordinate descent algorithm , 2013, J. Mach. Learn. Res..

[12]  Tong Zhang,et al.  Accelerating Stochastic Gradient Descent using Predictive Variance Reduction , 2013, NIPS.

[13]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[14]  Deanna Needell,et al.  Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method , 2012, ArXiv.

[15]  Adrian S. Lewis,et al.  Randomized Methods for Linear Constraints: Convergence Rates and Conditioning , 2008, Math. Oper. Res..

[16]  D. Needell.  Randomized Kaczmarz solver for noisy linear systems , 2009, 0902.0958.

[17]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[18]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[19]  S. Kaczmarz.  Approximate solution of systems of linear equations , 1993 .

[20]  I. J. Schoenberg,et al.  The Relaxation Method for Linear Inequalities , 1954, Canadian Journal of Mathematics.