Random reordering in SOR-type methods

When iteratively solving linear systems $$By=b$$By=b with Hermitian positive semi-definite B, and in particular when solving least-squares problems for $$Ax=b$$Ax=b by reformulating them as $$AA^*y=b$$AA∗y=b, it is often observed that SOR type methods (Gauß-Seidel, Kaczmarz) perform suboptimally for the given equation ordering, and that random reordering improves the situation on average. This paper is an attempt to provide some additional theoretical support for this phenomenon. We show error bounds for two randomized versions, called shuffled and preshuffled SOR, that improve asymptotically upon the best known bounds for SOR with cyclic ordering. Our results are based on studying the behavior of the triangular truncation of Hermitian matrices with respect to their permutations.

[1]  Hans G. Feichtinger,et al.  Theory and practice of irregular sampling , 2021, Wavelets.

[2]  P. Casazza.  Consequences of the Marcus/Spielman/Srivastava Solution of the Kadison-Singer Problem , 2014, 1407.4768.

[3]  J. Bourgain,et al.  Invertibility of ‘large’ submatrices with applications to the geometry of Banach spaces and harmonic analysis , 1987 .

[4]  Richard S. Varga,et al.  Orderings of the successive overrelaxation scheme , 1959 .

[5]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[6]  Stephen J. Wright.  Coordinate descent algorithms , 2015, Mathematical Programming.

[7]  Christopher Ré,et al.  Toward a Noncommutative Arithmetic-geometric Mean Inequality: Conjectures, Case-studies, and Consequences , 2012, COLT.

[8]  D. Young.  Iterative methods for solving partial difference equations of elliptic type , 1954 .

[9]  Roy Mathias,et al.  The Hadamard Operator Norm of a Circulant and Applications , 1997 .

[10]  W. Hackbusch.  Iterative Solution of Large Sparse Systems of Equations , 1993 .

[11]  P. Oswald,et al.  Convergence analysis for Kaczmarz-type methods in a Hilbert space framework , 2015 .

[12]  P. Casazza,et al.  The Kadison–Singer Problem in mathematics and engineering , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[13]  P. Oswald.  On the convergence rate of SOR: A worst case estimate , 2005, Computing.

[14]  John C. Duchi,et al.  Commentary on \Towards a Noncommutative Arithmetic-Geometric Mean Inequality" by B. Recht and C. R e , 2012 .

[15]  D. Spielman,et al.  Interlacing Families II: Mixed Characteristic Polynomials and the Kadison-Singer Problem , 2013, 1306.3969.