LRR for Subspace Segmentation via Tractable Schatten- $p$ Norm Minimization and Factorization

Recently, nuclear norm-based low rank representation (LRR) methods have been popular in several applications, such as subspace segmentation. However, there exist two limitations: one is that nuclear norm as the relaxation of rank function will lead to the suboptimal solution since nuclear norm-based minimization subproblem tends to the over-relaxations of singular value elements and treats each of them equally; the other is that solving LRR problems may cause more time consumption due to involving singular value decomposition of the large scale matrix at each iteration. To overcome both disadvantages, this paper mainly considers two tractable variants of LRR: one is Schatten-<inline-formula> <tex-math notation="LaTeX">${p}$ </tex-math></inline-formula> norm minimization-based LRR (i.e., <inline-formula> <tex-math notation="LaTeX">${S}_{p}$ </tex-math></inline-formula>NM_LRR) and the other is Schatten-<inline-formula> <tex-math notation="LaTeX">${p}$ </tex-math></inline-formula> norm factorization-based LRR (i.e., <inline-formula> <tex-math notation="LaTeX">${S}_{p}$ </tex-math></inline-formula>NF_LRR) for <inline-formula> <tex-math notation="LaTeX">${p=1}$ </tex-math></inline-formula>, 2/3 and 1/2. By introducing two or more auxiliary variables in the constraints, the alternating direction method of multiplier (ADMM) with multiple updating variables can be devised to solve these variants of LRR. Furthermore, both computational complexity and convergence property are given to evaluate nonconvex multiblocks ADMM algorithms. Several experiments finally validate the efficacy and efficiency of our methods on both synthetic data and real world data.

[1]  Inderjit S. Dhillon,et al.  Matrix Completion with Noisy Side Information , 2015, NIPS.

[2]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[3]  Rama Chellappa,et al.  Learning Common and Feature-Specific Patterns: A Novel Multiple-Sparse-Representation-Based Tracker , 2018, IEEE Transactions on Image Processing.

[4]  Licheng Jiao,et al.  A fast tri-factorization method for low-rank matrix recovery and completion , 2013, Pattern Recognit..

[5]  Feiping Nie,et al.  Subspace Clustering via New Low-Rank Model with Discrete Group Structure Constraint , 2016, IJCAI.

[6]  Feiping Nie,et al.  Multi-Subspace Representation and Discovery , 2011, ECML/PKDD.

[7]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[8]  Feiping Nie,et al.  Unsupervised and semi-supervised learning via ℓ1-norm graph , 2011, 2011 International Conference on Computer Vision.

[9]  David Zhang,et al.  A Generalized Iterated Shrinkage Algorithm for Non-convex Sparse Coding , 2013, 2013 IEEE International Conference on Computer Vision.

[10]  S. Shankar Sastry,et al.  Generalized principal component analysis (GPCA) , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Feiping Nie,et al.  Low-Rank Matrix Recovery via Efficient Schatten p-Norm Minimization , 2012, AAAI.

[12]  Yuanyuan Liu,et al.  Scalable Algorithms for Tractable Schatten Quasi-Norm Minimization , 2016, AAAI.

[13]  Carlo Fischione,et al.  On the Convergence of Alternating Direction Lagrangian Methods for Nonconvex Structured Optimization Problems , 2014, IEEE Transactions on Control of Network Systems.

[14]  Jian Yang,et al.  Robust Image Regression Based on the Extended Matrix Variate Power Exponential Distribution of Dependent Noise , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Yin Zhang,et al.  Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm , 2012, Mathematical Programming Computation.

[16]  Shuicheng Yan,et al.  Generalized Singular Value Thresholding , 2014, AAAI.

[17]  Zhang Yi,et al.  Constructing the L2-Graph for Robust Subspace Learning and Subspace Clustering , 2012, IEEE Transactions on Cybernetics.

[18]  John Wright,et al.  Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices via Convex Optimization , 2009, NIPS.

[19]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[20]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[21]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[22]  Jian Yang,et al.  Weighted sparse coding regularized nonconvex matrix regression for robust face recognition , 2017, Inf. Sci..

[23]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[24]  Xuelong Li,et al.  Multi-view Subspace Clustering , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[25]  Zhi-Quan Luo,et al.  Bilinear Factor Matrix Norm Minimization for Robust PCA: Algorithms and Applications , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[27]  Takeo Kanade,et al.  A Multibody Factorization Method for Independently Moving Objects , 1998, International Journal of Computer Vision.

[28]  Ying Zhang,et al.  Restricted $p$ -Isometry Properties of Nonconvex Matrix Recovery , 2013, IEEE Transactions on Information Theory.

[29]  Shuicheng Yan,et al.  A Unified Alternating Direction Method of Multipliers by Majorization Minimization , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[31]  G. Lerman,et al.  Robust recovery of multiple subspaces by geometric l_p minimization , 2011, 1104.3770.

[32]  Leon Wenliang Zhong,et al.  Fast Low-Rank Matrix Learning with Nonconvex Regularization , 2015, 2015 IEEE International Conference on Data Mining.

[33]  Maja Pantic,et al.  Dynamic Behavior Analysis via Structured Rank Minimization , 2017, International Journal of Computer Vision.

[34]  Feiping Nie,et al.  The Constrained Laplacian Rank Algorithm for Graph-Based Clustering , 2016, AAAI.

[35]  Shiqian Ma,et al.  On the Global Linear Convergence of the ADMM with MultiBlock Variables , 2014, SIAM J. Optim..

[36]  David J. Kriegman,et al.  Clustering appearances of objects under varying illumination conditions , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[37]  René Vidal,et al.  Combined central and subspace clustering for computer vision applications , 2006, ICML.

[38]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[39]  Qingshan Liu,et al.  Blessing of Dimensionality: Recovering Mixture Data via Dictionary Pursuit , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Hongchuan Yu,et al.  Constrained Low-Rank Representation for Robust Subspace Clustering , 2017, IEEE Transactions on Cybernetics.

[41]  Stephen P. Boyd,et al.  A rank minimization heuristic with application to minimum order system approximation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[42]  Zongben Xu,et al.  Enhancing Low-Rank Subspace Clustering by Manifold Regularization , 2014, IEEE Transactions on Image Processing.

[43]  Renato D. C. Monteiro,et al.  Local Minima and Convergence in Low-Rank Semidefinite Programming , 2005, Math. Program..

[44]  Zhang Yi,et al.  Connections Between Nuclear-Norm and Frobenius-Norm-Based Representations , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Feiping Nie,et al.  Joint Schatten $$p$$p-norm and $$\ell _p$$ℓp-norm robust matrix completion for missing value recovery , 2013, Knowledge and Information Systems.

[46]  Allen Y. Yang,et al.  Unsupervised segmentation of natural images via lossy data compression , 2008, Comput. Vis. Image Underst..

[47]  René Vidal,et al.  Multiframe Motion Segmentation with Missing Data Using PowerFactorization and GPCA , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[48]  Feiping Nie,et al.  Clustering and projected clustering with adaptive neighbors , 2014, KDD.

[49]  Zongben Xu,et al.  Fast image deconvolution using closed-form thresholding formulas of regularization , 2013, J. Vis. Commun. Image Represent..

[50]  Lei Zhang,et al.  Weighted Nuclear Norm Minimization and Its Applications to Low Level Vision , 2016, International Journal of Computer Vision.

[51]  Zhixun Su,et al.  Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation , 2011, NIPS.

[52]  Shuicheng Yan,et al.  Generalized Nonconvex Nonsmooth Low-Rank Minimization , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Qingshan Liu,et al.  A Deterministic Analysis for LRR , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Shuicheng Yan,et al.  Robust and Efficient Subspace Segmentation via Least Squares Regression , 2012, ECCV.

[55]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[56]  Yuanyuan Liu,et al.  Unified Scalable Equivalent Formulations for Schatten Quasi-Norms , 2016, ArXiv.

[57]  Rama Chellappa,et al.  Joint Sparse Representation and Robust Feature-Level Fusion for Multi-Cue Visual Tracking , 2015, IEEE Transactions on Image Processing.

[58]  J. Friedman.  Fast sparse regression and classification , 2012 .

[59]  Martin Jaggi,et al.  A Simple Algorithm for Nuclear Norm Regularized Problems , 2010, ICML.

[60]  Licheng Jiao,et al.  An efficient matrix factorization based low-rank representation for subspace clustering , 2013, Pattern Recognit..

[61]  Zhang Yi,et al.  A Unified Framework for Representation-Based Subspace Clustering of Out-of-Sample and Large-Scale Data , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[62]  Zhihua Zhang,et al.  Nonconvex Relaxation Approaches to Robust Matrix Recovery , 2013, IJCAI.

[63]  René Vidal,et al.  Low rank subspace clustering (LRSC) , 2014, Pattern Recognit. Lett..

[64]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[65]  Shuicheng Yan,et al.  Smoothed Low Rank and Sparse Matrix Recovery by Iteratively Reweighted Least Squares Minimization , 2014, IEEE Transactions on Image Processing.

[66]  Yong Yu,et al.  Robust Recovery of Subspace Structures by Low-Rank Representation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Junfeng Yang,et al.  Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization , 2012, Math. Comput..

[68]  Chen Xu,et al.  Schatten-q regularizer constrained low rank subspace clustering model , 2016, Neurocomputing.

[69]  Yuanyuan Liu,et al.  Tractable and Scalable Schatten Quasi-Norm Approximations for Rank Minimization , 2016, AISTATS.

[70]  Jiwen Lu,et al.  Automatic Subspace Learning via Principal Coefficients Embedding , 2014, IEEE Transactions on Cybernetics.

[71]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[72]  Feiping Nie,et al.  Robust Matrix Completion via Joint Schatten p-Norm and lp-Norm Minimization , 2012, 2012 IEEE 12th International Conference on Data Mining.

[73]  Tommi S. Jaakkola,et al.  Maximum-Margin Matrix Factorization , 2004, NIPS.

[74]  Thierry Bouwmans,et al.  Traditional and recent approaches in background modeling for foreground detection: An overview , 2014, Comput. Sci. Rev..

[75]  Mikael Skoglund,et al.  Performance guarantees for Schatten-p quasi-norm minimization in recovery of low-rank matrices , 2014, Signal Process..

[76]  Hristo S. Sendov,et al.  Nonsmooth Analysis of Singular Values. Part I: Theory , 2005 .

[77]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .