Unifying Matrix Data Structures: Simplifying and Speeding up Iterative Algorithms

Many algorithms use data structures that maintain properties of matrices undergoing some changes. The applications are wide-ranging and include for example matchings, shortest paths, linear programming, semi-definite programming, convex hull and volume computation. Given the wide range of applications, the exact property these data structures must maintain varies from one application to another, forcing algorithm designers to invent them from scratch or modify existing ones. Thus it is not surprising that these data structures and their proofs are usually tailor-made for their specific application and that maintaining more complicated properties results in more complicated proofs. In this paper we present a unifying framework that captures a wide range of these data structures. The simplicity of this framework allows us to give short proofs for many existing data structures regardless of how complicated the to be maintained property is. We also show how the framework can be used to speed up existing iterative algorithms, such as the simplex algorithm. More formally, consider any rational function $f(A_1,...,A_d)$ with input matrices $A_1,...,A_d$. We show that the task of maintaining $f(A_1,...,A_d)$ under updates to $A_1,...,A_d$ can be reduced to the much simpler problem of maintaining some matrix inverse $M^{-1}$ under updates to $M$. The latter is a well studied problem called dynamic matrix inverse. By applying our reduction and using known algorithms for dynamic matrix inverse we can obtain fast data structures and iterative algorithms for much more general problems.

[1]  Uri Zwick,et al.  An Improved Version of the Random-Facet Pivoting Rule for the Simplex Algorithm , 2015, STOC.

[2]  Piotr Sankowski,et al.  Subquadratic Algorithm for Dynamic Shortest Distances , 2005, COCOON.

[3]  Piotr Sankowski,et al.  Maximum matchings in planar graphs via gaussian elimination , 2004, Algorithmica.

[4]  Salil P. Vadhan,et al.  Computational Complexity , 2005, Encyclopedia of Cryptography and Security.

[5]  Gene H. Golub,et al.  The simplex method of linear programming using LU decomposition , 1969, Commun. ACM.

[6]  Volker Strassen,et al.  Algebraic Complexity Theory , 1991, Handbook of Theoretical Computer Science, Volume A: Algorithms and Complexity.

[7]  Danupon Nanongkai,et al.  Dynamic Approximate Shortest Paths and Beyond: Subquadratic and Worst-Case Update Time , 2019, 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS).

[8]  Yin Tat Lee,et al.  Solving Empirical Risk Minimization in the Current Matrix Multiplication Time , 2019, COLT.

[9]  Shang-Hua Teng,et al.  Smoothed analysis of algorithms: why the simplex algorithm usually takes polynomial time , 2001, STOC '01.

[10]  Piotr Sankowski,et al.  Dynamic transitive closure via dynamic matrix inverse: extended abstract , 2004, 45th Annual IEEE Symposium on Foundations of Computer Science.

[11]  Vissarion Fisikopoulos,et al.  Faster Geometric Algorithms via Dynamic Determinant Computation , 2012, ESA.

[12]  Richard Peng,et al.  Iterative Refinement for ℓp-norm Regression , 2019, SODA.

[13]  Daniel A. Spielman,et al.  Improved smoothed analysis of the shadow vertex simplex method , 2005, 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS'05).

[14]  Arne Storjohann,et al.  A Relaxed Algorithm for Online Matrix Inversion , 2015, ISSAC.

[15]  R. Plackett Some theorems in least squares. , 1950, Biometrika.

[16]  Yin Tat Lee,et al.  Solving linear programs in the current matrix multiplication time , 2018, STOC.

[17]  Omri Weinstein,et al.  Faster Dynamic Matrix Inverse for Faster LPs , 2020, ArXiv.

[18]  Peter A. Beling,et al.  Using Fast Matrix Multiplication to Find Basic Solutions , 1998, Theoretical Computer Science.

[19]  Thatchaphol Saranurak,et al.  Dynamic Matrix Inverse: Improved Algorithms and Matching Conditional Lower Bounds , 2019, 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS).

[20]  Gil Kalai,et al.  A subexponential randomized simplex algorithm (extended abstract) , 1992, STOC '92.

[21]  Narendra Karmarkar,et al.  A new polynomial-time algorithm for linear programming , 1984, Comb..

[22]  Yin Tat Lee,et al.  An improved cutting plane method for convex optimization, convex-concave games, and its applications , 2020, STOC.

[23]  Pravin M. Vaidya,et al.  Speeding-up linear programming using fast matrix multiplication , 1989, 30th Annual Symposium on Foundations of Computer Science.

[24]  Zhao Song,et al.  Solving tall dense linear programs in nearly linear time , 2020, STOC.

[25]  Yin Tat Lee,et al.  A Faster Cutting Plane Method and its Implications for Combinatorial and Convex Optimization , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[26]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[27]  James Demmel,et al.  Fast linear algebra is stable , 2006, Numerische Mathematik.

[28]  Richard Peng,et al.  Iterative Refinement for $\ell_p$-norm Regression , 2019, SODA 2019.

[29]  Erik Elmroth,et al.  Applying recursion to serial and parallel QR factorization leads to better performance , 2000, IBM J. Res. Dev..

[30]  Simon Lucey,et al.  Learning Background-Aware Correlation Filters for Visual Tracking , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Tim D. Barfoot,et al.  Online visual motion estimation using FastSLAM with SIFT features , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Thatchaphol Saranurak,et al.  Sensitive Distance and Reachability Oracles for Large Batch Updates , 2019, 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS).

[33]  Donald Goldfarb,et al.  On the Bartels—Golub decomposition for linear programming bases , 1977, Math. Program..

[34]  John K. Reid,et al.  A sparsity-exploiting variant of the Bartels—Golub decomposition for linear programming bases , 1982, Math. Program..

[35]  Daniel A. Spielman,et al.  A randomized polynomial-time simplex algorithm for linear programming , 2006, STOC '06.

[36]  Daniel Dadush,et al.  A friendly smoothed analysis of the simplex method , 2017, STOC.

[37]  Piotr Sankowski,et al.  Faster dynamic matchings and vertex connectivity , 2007, SODA '07.

[38]  R. Shamir The Efficiency of the Simplex Method: A Survey , 1987 .

[39]  R. Bartels A NUMERICAL INVESTIGATION OF THE SIMPLEX METHOD , 1968 .

[40]  V. Klee,et al.  HOW GOOD IS THE SIMPLEX ALGORITHM , 1970 .

[41]  Roman Vershynin,et al.  Beyond Hirsch Conjecture: Walks on Random Polytopes and Smoothed Complexity of the Simplex Method , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[42]  W. Morven Gentleman,et al.  Basic Procedures for Large, Sparse or Weighted Linear Least Squares Problems , 1974 .

[43]  Yin Tat Lee,et al.  Efficient Inverse Maintenance and Faster Algorithms for Linear Programming , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.