On $l_q$ Optimization and Matrix Completion

Rank minimization problems, which consist of finding a matrix of minimum rank subject to linear constraints, have been proposed in many areas of engineering and science. A specific problem is the matrix completion problem in which a low rank data matrix can be recovered from incomplete samples of its entries by solving a rank penalized least squares problem. The rank penalty is in fact the <i>l</i><sub>0</sub> “norm” of the matrix singular values. A recent convex relaxation of this penalty is the commonly used <i>l</i><sub>1</sub> norm of the matrix singular values. In this paper, we bridge the gap between these two penalties and propose the <i>lq</i>, 0 <; <i>q</i> <; 1 penalized least squares problem for matrix completion. An iterative algorithm is developed by solving a non-standard optimization problem and a non-trivial convergence result is proved. We illustrate with simulations comparing the reconstruction quality of the three matrix singular value penalty functions: <i>l</i><sub>0</sub>, <i>l</i><sub>1</sub>, and <i>lq</i>, 0 <; <i>q</i> <; 1.

[1]  Y. Ye,et al.  Lower Bound Theory of Nonzero Entries in Solutions of ℓ2-ℓp Minimization , 2010, SIAM J. Sci. Comput..

[2]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[3]  J. Pesquet,et al.  Wavelet thresholding for some classes of non–Gaussian noise , 2002 .

[4]  Jieping Ye,et al.  An accelerated gradient method for trace norm minimization , 2009, ICML '09.

[5]  Robert Tibshirani,et al.  Spectral Regularization Algorithms for Learning Large Incomplete Matrices , 2010, J. Mach. Learn. Res..

[6]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[7]  Kilian Q. Weinberger,et al.  Unsupervised Learning of Image Manifolds by Semidefinite Programming , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[8]  Shiqian Ma,et al.  Fixed point and Bregman iterative methods for matrix rank minimization , 2009, Math. Program..

[9]  Noga Alon,et al.  Generalization Error Bounds for Collaborative Prediction with Low-Rank Matrices , 2004, NIPS.

[10]  Michael Ruogu Zhang,et al.  Comprehensive identification of cell cycle-regulated genes of the yeast Saccharomyces cerevisiae by microarray hybridization. , 1998, Molecular biology of the cell.

[11]  José M. Bioucas-Dias,et al.  Adaptive total variation image deblurring: A majorization-minimization approach , 2009, Signal Process..

[12]  P. Huard,et al.  Point-to-set maps and mathematical programming , 1979 .

[13]  Nathan Srebro,et al.  Fast maximum margin matrix factorization for collaborative prediction , 2005, ICML.

[14]  Robert D. Nowak,et al.  Online identification and tracking of subspaces from highly incomplete information , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[15]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[16]  R. Chartrand,et al.  Restricted isometry properties and nonconvex compressive sensing , 2007 .

[17]  Olgica Milenkovic,et al.  A Geometric Approach to Low-Rank Matrix Completion , 2010, IEEE Transactions on Information Theory.

[18]  R. Tibshirani,et al.  PATHWISE COORDINATE OPTIMIZATION , 2007, 0708.1485.

[19]  Andrea Montanari,et al.  Matrix completion from a few entries , 2009, 2009 IEEE International Symposium on Information Theory.

[20]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[21]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[22]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[23]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[24]  T. Blumensath,et al.  Iterative Thresholding for Sparse Approximations , 2008 .

[25]  Stephen J. Wright,et al.  Sparse Reconstruction by Separable Approximation , 2008, IEEE Transactions on Signal Processing.

[26]  J. Friedman.  Fast sparse regression and classification , 2012 .

[27]  Rabab Kreidieh Ward,et al.  Some empirical advances in matrix completion , 2011, Signal Process..

[28]  Ben Taskar,et al.  Joint covariate selection and joint subspace selection for multiple classification problems , 2010, Stat. Comput..

[29]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[30]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[31]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[32]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[33]  Mathematical programming study 10 , 1979 .

[34]  Massimiliano Pontil,et al.  Convex multi-task feature learning , 2008, Machine Learning.

[35]  Pierre-Antoine Absil,et al.  RTRMC: A Riemannian trust-region method for low-rank matrix completion , 2011, NIPS.

[36]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[37]  D. Goldfarb,et al.  Solving low-rank matrix completion problems efficiently , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).