An Analysis of Sketched IRLS for Accelerated Sparse Residual Regression

This paper studies the problem of sparse residual regression, i.e., learning a linear model using a norm that favors solutions in which the residuals are sparsely distributed. This is a common problem in a wide range of computer vision applications where a linear system has a lot more equations than unknowns and we wish to find the maximum feasible set of equations by discarding unreliable ones. We show that one of the most popular solution methods, iteratively reweighted least squares (IRLS), can be significantly accelerated by the use of matrix sketching. We analyze the convergence behavior of the proposed method and show its efficiency on a range of computer vision applications. The source code for this project can be found at https://github.com/Diwata0909/Sketched_IRLS.

[1]  David P. Woodruff,et al.  Low rank approximation and regression in input sparsity time , 2012, STOC '13.

[2]  Peter Richtárik,et al.  Randomized Iterative Methods for Linear Systems , 2015, SIAM J. Matrix Anal. Appl..

[3]  David Zhang,et al.  A Survey of Sparse Representation: Algorithms and Applications , 2015, IEEE Access.

[4]  Joel A. Tropp,et al.  ALGORITHMS FOR SIMULTANEOUS SPARSE APPROXIMATION , 2006 .

[5]  Hassan Foroosh,et al.  Initialized iterative reweighted least squares for automatic target recognition , 2015, MILCOM 2015 - 2015 IEEE Military Communications Conference.

[6]  Shuicheng Yan,et al.  Smoothed Low Rank and Sparse Matrix Recovery by Iteratively Reweighted Least Squares Minimization , 2014, IEEE Transactions on Image Processing.

[7]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[8]  Juliane Sigl,et al.  Nonlinear residual minimization by iteratively reweighted least squares , 2015, Computational Optimization and Applications.

[9]  Tae-Hyun Oh,et al.  Fast Randomized Singular Value Thresholding for Low-Rank Optimization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  J. Neumann,et al.  Numerical inverting of matrices of high order. II , 1951 .

[11]  Frank Plastria,et al.  On the point for which the sum of the distances to n given points is minimum , 2009, Ann. Oper. Res..

[12]  Thomas Brox,et al.  On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex Optimization in Computer Vision , 2015, SIAM J. Imaging Sci..

[13]  Jakub W. Pachocki,et al.  Solving SDD linear systems in nearly mlog1/2n time , 2014, STOC.

[14]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Stéphane Mallat,et al.  A Wavelet Tour of Signal Processing - The Sparse Way, 3rd Edition , 2008 .

[16]  Simon French,et al.  Finite Algorithms in Optimization and Data Analysis , 1986 .

[17]  J. Neumann,et al.  Numerical inverting of matrices of high order , 1947 .

[18]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[19]  Jean-Luc Starck,et al.  Sparse Solution of Underdetermined Systems of Linear Equations by Stagewise Orthogonal Matching Pursuit , 2012, IEEE Transactions on Information Theory.

[20]  Takeo Kanade,et al.  Robust L/sub 1/ norm factorization in the presence of outliers and missing data by alternative convex programming , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[21]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[22]  Gérard G. Medioni,et al.  Object modeling by registration of multiple range images , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[23]  Stphane Mallat,et al.  A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way , 2008 .

[24]  S. Muthukrishnan,et al.  Sampling algorithms for l2 regression and applications , 2006, SODA '06.

[25]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[26]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[27]  Tom Drummond,et al.  Solving Robust Regularization Problems Using Iteratively Re-weighted Least Squares , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[28]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[29]  Allen Y. Yang,et al.  Fast L1-Minimization Algorithms For Robust Face Recognition , 2010, 1007.3753.

[30]  David P. Woodruff.  Sketching as a Tool for Numerical Linear Algebra , 2014, Found. Trends Theor. Comput. Sci..

[31]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[32]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[33]  Joel A. Tropp,et al.  Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit , 2006, Signal Process..

[34]  Junzhou Huang,et al.  Fast iteratively reweighted least squares algorithms for analysis‐based sparse reconstruction , 2018, Medical Image Anal..

[35]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[36]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[37]  Steven L. Brunton,et al.  Randomized Dynamic Mode Decomposition , 2017, SIAM J. Appl. Dyn. Syst..

[38]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[39]  Adrien Bartoli,et al.  Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces , 2013, BMVC.

[40]  Bernard Chazelle,et al.  Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform , 2006, STOC '06.

[41]  Michael W. Mahoney.  Randomized Algorithms for Matrices and Data , 2011, Found. Trends Mach. Learn..

[42]  Moses Charikar,et al.  Finding frequent items in data streams , 2002, Theor. Comput. Sci..

[43]  Jed A. Duersch,et al.  Randomized QR with Column Pivoting , 2015, SIAM J. Sci. Comput..

[44]  S. Muthukrishnan,et al.  Faster least squares approximation , 2007, Numerische Mathematik.

[45]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[46]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[47]  James E. Gentle,et al.  Matrix Algebra: Theory, Computations, and Applications in Statistics , 2007 .

[48]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[49]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[50]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[51]  Tamás Sarlós,et al.  Improved Approximation Algorithms for Large Matrices via Random Projections , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[52]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[53]  C GilbertAnna,et al.  Algorithms for simultaneous sparse approximation. Part II , 2006 .

[54]  M. Shirosaki.  Another proof of the defect relation for moving targets , 1991 .

[55]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[56]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[57]  Narendra Karmarkar,et al.  A new polynomial-time algorithm for linear programming , 1984, Comb..

[58]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[59]  Richard I. Hartley,et al.  Convergence of Iteratively Re-weighted Least Squares to Robust M-Estimators , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[60]  Robert M. Gower,et al.  Randomized Quasi-Newton Updates Are Linearly Convergent Matrix Inversion Algorithms , 2016, SIAM J. Matrix Anal. Appl..

[61]  Aritra Dutta,et al.  Online and Batch Supervised Background Estimation Via L1 Regression , 2017, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[62]  P WoodruffDavid.  Sketching as a Tool for Numerical Linear Algebra , 2014 .

[63]  Jochen Trumpf,et al.  Generalized Weiszfeld Algorithms for Lq Optimization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Joel A. Tropp,et al.  Improved Analysis of the subsampled Randomized Hadamard Transform , 2010, Adv. Data Sci. Adapt. Anal..

[66]  Georgios B. Giannakis,et al.  From Sparse Signals to Sparse Residuals for Robust Sensing , 2011, IEEE Transactions on Signal Processing.