Low-rank incremental methods for computing dominant singular subspaces☆

Computing the singular values and vectors of a matrix is a crucial kernel in numerous scientific and industrial applications. As such, numerous methods have been proposed to handle this problem in a computationally efficient way. This paper considers a family of methods for incrementally computing the dominant SVD of a large matrix A. Specifically, we describe a unification of a number of previously independent methods for approximating the dominant SVD after a single pass through A. We connect the behavior of these methods to that of a class of optimization-based iterative eigensolvers on ATA. An iterative procedure is proposed which allows the computation of an accurate dominant SVD using multiple passes through A. We present an analysis of the convergence of this iteration and provide empirical demonstration of the proposed method on both synthetic and benchmark data.

[1]  Matthew Brand,et al.  Incremental Singular Value Decomposition of Uncertain Data with Missing Values , 2002, ECCV.

[2]  L. Sirovich.  Empirical Eigenfunctions and Low Dimensional Systems , 1991 .

[3]  Danny C. Sorensen,et al.  A Symmetry Preserving Singular Value Decomposition , 2006, SIAM J. Matrix Anal. Appl..

[4]  Z. Drmač,et al.  A new stable bidiagonal reduction algorithm , 2005 .

[5]  L. Sirovich.  Turbulence and the dynamics of coherent structures. I. Coherent structures , 1987 .

[6]  L Sirovich,et al.  Low-dimensional procedure for the characterization of human faces. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[7]  Christopher Grover Baker.  Block Incremental Algorithm for Computing Dominant Singular Subspaces , 2004 .

[8]  J. Bunch,et al.  Updating the singular value decomposition , 1978 .

[9]  Michael Lindenbaum,et al.  Sequential Karhunen-Loeve basis extraction and its application to images , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[10]  G. Stewart.  Matrix Algorithms, Volume II: Eigensystems , 2001 .

[11]  Petros Drineas,et al.  Fast Monte Carlo Algorithms for Matrices III: Computing a Compressed Approximate Matrix Decomposition , 2006, SIAM J. Comput..

[12]  Lars Elden,et al.  Matrix methods in data mining and pattern recognition , 2007, Fundamentals of algorithms.

[13]  G. W. Stewart,et al.  Matrix Algorithms: Volume 1, Basic Decompositions , 1998 .

[14]  Michiel E. Hochstenbach,et al.  A Jacobi-Davidson Type SVD Method , 2001, SIAM J. Sci. Comput..

[15]  Chao Yang,et al.  ARPACK users' guide - solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods , 1998, Software, environments, tools.

[16]  B. S. Manjunath,et al.  An Eigenspace Update Algorithm for Image Analysis , 1997, CVGIP Graph. Model. Image Process..

[17]  Paul Van Dooren,et al.  Recursive Calculation of Dominant Singular Subspaces , 2003, SIAM J. Matrix Anal. Appl..

[18]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[19]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[20]  Ricardo C. H. del Rosario,et al.  Proper orthogonal decomposition-based control of transverse beam vibrations: experimental implementation , 2002, IEEE Trans. Control. Syst. Technol..

[21]  Gene H. Golub,et al.  Matrix computations , 1983 .

[22]  Matthew Brand,et al.  Fast Online SVD Revisions for Lightweight Recommender Systems , 2003, SDM.

[23]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[24]  Jang-Gyu Lee,et al.  On updating the singular value decomposition , 1996, Proceedings of International Conference on Communication Technology. ICCT '96.

[25]  Paul Van Dooren,et al.  An incremental method for computing dominant singular spaces , 2001 .

[26]  M. Brand,et al.  Fast low-rank modifications of the thin singular value decomposition , 2006 .

[27]  Christian H. Bischof,et al.  The WY representation for products of householder matrices , 1985, PPSC.

[28]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[29]  C. Loan,et al.  A Storage-Efficient $WY$ Representation for Products of Householder Transformations , 1989 .

[30]  P. Holmes,et al.  The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows , 1993 .

[31]  Michael J. O'Hara.  On Low-Rank Updates to the Singular Value and Tucker Decompositions , 2010, SDM.

[32]  Ke Wang,et al.  Proceedings of the SIAM International Conference on Data Mining, SDM 2009, April 30 - May 2, 2009, Sparks, Nevada, USA , 2009, SDM.

[33]  Hasan Erbay,et al.  Modifiable low‐rank approximation to a matrix , 2009, Numer. Linear Algebra Appl..

[34]  Lawrence Sirovich,et al.  New Perspectives in Turbulence , 2011 .

[35]  Pierre-Antoine Absil,et al.  Trust-Region Methods on Riemannian Manifolds , 2007, Found. Comput. Math..

[36]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[37]  Allen A. Goldstein,et al.  Constructive Real Analysis , 1967 .