A Proximal Alternating Direction Method for Semi-Definite Rank Minimization

Semi-definite rank minimization problems model a wide range of applications in both signal processing and machine learning fields. This class of problem is NP-hard in general. In this paper, we propose a proximal Alternating Direction Method (ADM) for the well-known semi-definite rank regularized minimization problem. Specifically, we first reformulate this NP-hard problem as an equivalent biconvex MPEC (Mathematical Program with Equilibrium Constraints), and then solve it using proximal ADM, which involves solving a sequence of structured convex semi-definite subproblems to find a desirable solution to the original rank regularized optimization problem. Moreover, based on the Kurdyka-?ojasiewicz inequality, we prove that the proposed method always converges to a KKT stationary point under mild conditions. We apply the proposed method to the widely studied and popular sensor network localization problem. Our extensive experiments demonstrate that the proposed algorithm outperforms state-of-the-art low-rank semi-definite minimization algorithms in terms of solution quality.

[1]  Anthony Man-Cho So,et al.  Theory of semidefinite programming for Sensor Network Localization , 2005, SODA '05.

[2]  Feiping Nie,et al.  Low-Rank Matrix Recovery via Efficient Schatten p-Norm Minimization , 2012, AAAI.

[3]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[4]  Jon C. Dattorro,et al.  Convex Optimization & Euclidean Distance Geometry , 2004 .

[5]  Yong Zhang,et al.  Sparse Approximation via Penalty Decomposition Methods , 2012, SIAM J. Optim..

[6]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[7]  Narendra Ahuja,et al.  Robust visual tracking via multi-task sparse learning , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Bethany L. Nicholson,et al.  Mathematical Programs with Equilibrium Constraints , 2021, Pyomo — Optimization Modeling in Python.

[9]  Kim-Chuan Toh,et al.  Semidefinite Programming Approaches for Sensor Network Localization With Noisy Distance Measurements , 2006, IEEE Transactions on Automation Science and Engineering.

[10]  Vincent K. N. Lau,et al.  Rank-Constrained Schur-Convex Optimization With Multiple Trace/Log-Det Constraints , 2011, IEEE Transactions on Signal Processing.

[11]  Bernard Ghanem,et al.  ℓ0TV: A new method for image restoration in the presence of impulse noise , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Paul Tseng,et al.  (Robust) Edge-based semidefinite programming relaxation of sensor network localization , 2011, Math. Program..

[13]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[14]  Zhaosong Lu,et al.  Penalty decomposition methods for rank minimization , 2010, Optim. Methods Softw..

[15]  Anthony Man-Cho So,et al.  Beyond convex relaxation: A polynomial-time non-convex optimization approach to network localization , 2013, 2013 Proceedings IEEE INFOCOM.

[16]  Yinyu Ye,et al.  Semidefinite programming based algorithms for sensor network localization , 2006, TOSN.

[17]  Xiaolan Liu,et al.  Exact Penalty Decomposition Method for Zero-Norm Minimization Based on MPEC Formulation , 2014, SIAM J. Sci. Comput..

[18]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[19]  Eilyan Bitar,et al.  A rank minimization algorithm to enhance semidefinite relaxations of Optimal Power Flow , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[20]  Qionghai Dai,et al.  Low-Rank Structure Learning via Nonconvex Heuristic Recovery , 2010, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Ligang Liu,et al.  An as-rigid-as-possible approach to sensor network localization , 2010, TOSN.

[22]  Zhixun Su,et al.  Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation , 2011, NIPS.

[23]  Stephen P. Boyd,et al.  Further Relaxations of the Semidefinite Programming Approach to Sensor Network Localization , 2008, SIAM J. Optim..

[24]  LuZhaosong.  Iterative reweighted minimization methods for $$l_p$$lp regularized unconstrained nonlinear programming , 2014 .

[25]  Renato D. C. Monteiro,et al.  Digital Object Identifier (DOI) 10.1007/s10107-004-0564-1 , 2004 .

[26]  Yin Zhang,et al.  Exploiting temporal stability and low-rank structure for localization in mobile networks , 2010, MobiCom.

[27]  Shuicheng Yan,et al.  Generalized Singular Value Thresholding , 2014, AAAI.

[28]  Shuicheng Yan,et al.  Generalized Nonconvex Nonsmooth Low-Rank Minimization , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Changsheng Xu,et al.  Low-Rank Sparse Coding for Image Classification , 2013, 2013 IEEE International Conference on Computer Vision.

[30]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[31]  Xuelong Li,et al.  Fast and Accurate Matrix Completion via Truncated Nuclear Norm Regularization , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Nathan Krislock,et al.  Explicit Sensor Network Localization using Semidefinite Representations and Facial Reductions , 2010, SIAM J. Optim..

[33]  Defeng Sun,et al.  A rank-corrected procedure for matrix completion with fixed basis coefficients , 2012, Math. Program..

[34]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[35]  Yun-Bin Zhao,et al.  Approximation Theory of Matrix Rank Minimization and Its Application to Quadratic Equations , 2010, 1010.0851.

[36]  Paul Tseng,et al.  Second-Order Cone Programming Relaxation of Sensor Network Localization , 2007, SIAM J. Optim..

[37]  Jiebo Luo,et al.  Self-Supervised Online Metric Learning With Low Rank Constraint for Scene Categorization , 2013, IEEE Transactions on Image Processing.

[38]  Stephen P. Boyd,et al.  Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices , 2003, Proceedings of the 2003 American Control Conference, 2003..

[39]  Rongrong Ji,et al.  Low-Rank Similarity Metric Learning in High Dimensions , 2015, AAAI.

[40]  Houduo Qi,et al.  A Sequential Semismooth Newton Method for the Nearest Low-rank Correlation Matrix Problem , 2011, SIAM J. Optim..

[41]  Chinmay Hegde,et al.  NuMax: A Convex Approach for Learning Near-Isometric Linear Embeddings , 2015, IEEE Transactions on Signal Processing.

[42]  Gábor Pataki,et al.  On the Rank of Extreme Matrices in Semidefinite Programs and the Multiplicity of Optimal Eigenvalues , 1998, Math. Oper. Res..

[43]  Daniel Pérez Palomar,et al.  Rank-Constrained Separable Semidefinite Programming With Applications to Optimal Beamforming , 2010, IEEE Transactions on Signal Processing.

[44]  David J. Kriegman,et al.  Generalized Non-metric Multidimensional Scaling , 2007, AISTATS.

[45]  Matthieu Cord,et al.  Fantope Regularization in Metric Learning , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Renato D. C. Monteiro,et al.  A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization , 2003, Math. Program..

[47]  Yinyu Ye,et al.  Semidefinite programming for ad hoc wireless sensor network localization , 2004, Third International Symposium on Information Processing in Sensor Networks, 2004. IPSN 2004.

[48]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[49]  Hédy Attouch,et al.  On the convergence of the proximal algorithm for nonsmooth functions involving analytic features , 2008, Math. Program..

[50]  Inderjit S. Dhillon,et al.  Rank minimization via online learning , 2008, ICML '08.

[51]  Pablo A. Parrilo,et al.  Diagonal and Low-Rank Matrix Decompositions, Correlation Matrices, and Ellipsoid Fitting , 2012, SIAM J. Matrix Anal. Appl..

[52]  Wotao Yin,et al.  A Block Coordinate Descent Method for Regularized Multiconvex Optimization with Applications to Nonnegative Tensor Factorization and Completion , 2013, SIAM J. Imaging Sci..