Penalty decomposition methods for rank minimization

In this paper we consider general rank minimization problems with rank appearing either in the objective function or as a constraint. We first establish that a class of special rank minimization problems has closed-form solutions. Using this result, we then propose penalty decomposition (PD) methods for general rank minimization problems in which each subproblem is solved by a block coordinate descent method. Under some suitable assumptions, we show that any accumulation point of the sequence generated by the PD methods satisfies the first-order optimality conditions of a nonlinear reformulation of the problems. Finally, we test the performance of our methods by applying them to the matrix completion and nearest low-rank correlation matrix problems. The computational results demonstrate that our methods are generally comparable or superior to the existing methods in terms of solution quality.

[1]  R. Tibshirani,et al.  Regularization methods for learning incomplete matrices , 2009, 0906.2034.

[2]  Arkadi Nemirovski,et al.  Lectures on modern convex optimization - analysis, algorithms, and engineering applications , 2001, MPS-SIAM series on optimization.

[3]  Michael P. Friedlander,et al.  Sparse Optimization with Least-Squares Constraints , 2011, SIAM J. Optim..

[4]  Anthony Man-Cho So,et al.  Beyond convex relaxation: A polynomial-time non-convex optimization approach to network localization , 2013, 2013 Proceedings IEEE INFOCOM.

[5]  John Riedl,et al.  An Algorithmic Framework for Performing Collaborative Filtering , 1999, SIGIR Forum.

[6]  Massimo Fornasier,et al.  Low-rank Matrix Recovery via Iteratively Reweighted Least Squares Minimization , 2010, SIAM J. Optim..

[7]  Yong-Jin Liu,et al.  An implementable proximal point algorithmic framework for nuclear norm minimization , 2012, Math. Program..

[8]  R. Bhatia.  Matrix Analysis , 1996 .

[9]  Ying Xiong.  Nonlinear Optimization , 2014 .

[10]  D. Brigo,et al.  Interest Rate Models , 2001 .

[11]  Sewoong Oh,et al.  A Gradient Descent Algorithm on the Grassman Manifold for Matrix Completion , 2009, ArXiv.

[12]  Defeng Sun,et al.  A Majorized Penalty Approach for Calibrating Rank Constrained Correlation Matrix Problems , 2010 .

[13]  Lars Elden,et al.  Matrix methods in data mining and pattern recognition , 2007, Fundamentals of algorithms.

[14]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[15]  Yoram Bresler,et al.  ADMiRA: Atomic Decomposition for Minimum Rank Approximation , 2009, IEEE Transactions on Information Theory.

[16]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[17]  H. Qi,et al.  An augmented Lagrangian dual approach for the H-weighted nearest correlation matrix problem , 2011 .

[18]  Igor Grubisic,et al.  Efficient Rank Reduction of Correlation Matrices , 2004, cond-mat/0403477.

[19]  Lixin Wu.  Fast at-the-money calibration of the LIBOR market model through Lagrange multipliers , 2002 .

[20]  Wotao Yin,et al.  Improved Iteratively Reweighted Least Squares for Unconstrained Smoothed 퓁q Minimization , 2013, SIAM J. Numer. Anal..

[21]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[22]  A. Ruszczynski,et al.  Nonlinear Optimization , 2006 .

[23]  M. Fazel,et al.  Iterative reweighted least squares for matrix rank minimization , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[24]  R. Rebonato.  Review Paper. Interest–rate term–structure pricing models: a review , 2004, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[25]  P. Tseng.  Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization , 2001 .

[26]  Nathan Srebro,et al.  Fast maximum margin matrix factorization for collaborative prediction , 2005, ICML.

[27]  Shiqian Ma,et al.  Fixed point and Bregman iterative methods for matrix rank minimization , 2009, Math. Program..

[28]  Stephen P. Boyd,et al.  A rank minimization heuristic with application to minimum order system approximation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[29]  David P. Williamson,et al.  .879-approximation algorithms for MAX CUT and MAX 2SAT , 1994, STOC '94.

[30]  Olgica Milenkovic,et al.  SET: An algorithm for consistent matrix completion , 2009, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[31]  Riccardo Rebonato.  Modern pricing of interest-rate derivatives , 2002 .

[32]  Miriam Hodge,et al.  Fast at-the-money calibration of the Libor market model using Lagrange multipliers , 2003 .

[33]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[34]  P. Groenen,et al.  Rank Reduction of Correlation Matrices by Majorization , 2004 .

[35]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[36]  Yin Zhang,et al.  Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm , 2012, Mathematical Programming Computation.

[37]  Zhaosong Lu,et al.  Iterative Reweighted Singular Value Minimization Methods for $l_p$ Regularized Unconstrained Matrix Minimization , 2014, ArXiv.

[38]  Takeo Kanade,et al.  A sequential factorization method for recovering shape and motion from image streams , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Renato D. C. Monteiro,et al.  Convex optimization methods for dimension reduction and coefficient estimation in multivariate linear regression , 2009, Mathematical Programming.

[40]  A. S. Lewis,et al.  Derivatives of Spectral Functions , 1996, Math. Oper. Res..

[41]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[42]  Lieven Vandenberghe,et al.  Interior-Point Method for Nuclear Norm Approximation with Application to System Identification , 2009, SIAM J. Matrix Anal. Appl..

[43]  Tadayoshi Fushiki,et al.  Estimation of Positive Semidefinite Correlation Matrices by Using Convex Quadratic Semidefinite Programming , 2009, Neural Computation.

[44]  Robert Tibshirani,et al.  Spectral Regularization Algorithms for Learning Large Incomplete Matrices , 2010, J. Mach. Learn. Res..

[45]  Guangdong Feng,et al.  A Tensor Based Method for Missing Traffic Data Completion , 2013 .

[46]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[47]  Patrick J F Groenen,et al.  Rank reduction of correlation matrices by majorization , 2004 .

[48]  Houduo Qi,et al.  A Sequential Semismooth Newton Method for the Nearest Low-rank Correlation Matrix Problem , 2011, SIAM J. Optim..

[49]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[50]  Riccardo Rebonato,et al.  On the simultaneous calibration of multifactor lognormal interest rate models to Black volatilities and to the correlation matrix , 1999 .

[51]  Yin Zhang,et al.  Maximum stable set formulations and heuristics based on continuous optimization , 2002, Math. Program..

[52]  Zhenyue Zhang,et al.  Optimal low-rank approximation to a correlation matrix , 2003 .

[53]  D. Brigo,et al.  A Note on Correlation and Rank Reduction , 2002 .

[54]  Zhaosong Lu,et al.  Penalty Decomposition Methods for $L0$-Norm Minimization , 2010, ArXiv.

[55]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[56]  Inderjit S. Dhillon,et al.  Guaranteed Rank Minimization via Singular Value Projection , 2009, NIPS.

[57]  D. Brigo,et al.  Parameterizing correlations: a geometric interpretation , 2007 .