Linearized Alternating Direction Method of Multipliers for Constrained Nonconvex Regularized Optimization

In this paper, we consider a wide class of constrained nonconvex regularized minimization problems, where the constraints are linearly constraints. It was reported in the literature that nonconvex regularization usually yields a solution with more desirable sparse structural properties beyond convex ones. However, it is not easy to obtain the proximal mapping associated with nonconvex regularization, due to the imposed linearly constraints. In this paper, the optimization problem with linear constraints is solved by the Linearized Alternating Direction Method of Multipliers (LADMM). Moreover, we present a detailed convergence analysis of the LADMM algorithm for solving nonconvex compositely regularized optimization with a large class of nonconvex penalties. Experimental results on several real-world datasets validate the efficacy of the proposed algorithm.

[1]  Wusheng Luo,et al.  Compressed Sensing by Wavelet-Based Contourlet Transform , 2011, 2011 International Conference of Information Technology, Computer Engineering and Management Sciences.

[2]  Leon Wenliang Zhong,et al.  Fast Stochastic Alternating Direction Method of Multipliers , 2013, ICML.

[3]  Zhi-Quan Luo,et al.  Parallel Direction Method of Multipliers , 2014, NIPS.

[4]  Tong Zhang,et al.  Analysis of Multi-stage Convex Relaxation for Sparse Regularization , 2010, J. Mach. Learn. Res..

[5]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[6]  Wei Liu,et al.  Vehicle logo recognition by spatial-SIFT combined with logistic regression , 2016, 2016 19th International Conference on Information Fusion (FUSION).

[7]  Zongben Xu,et al.  Convergence of multi-block Bregman ADMM for nonconvex composite problems , 2015, Science China Information Sciences.

[8]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[9]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[10]  Shiqian Ma,et al.  Sparse Inverse Covariance Selection via Alternating Linearization Methods , 2010, NIPS.

[11]  Tong Zhang.  Multi-stage Convex Relaxation for Feature Selection , 2011, 1106.0565.

[12]  J. Toland.  A duality principle for non-convex optimisation and the calculus of variations , 1979 .

[13]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[14]  R. Tibshirani,et al.  The solution path of the generalized lasso , 2010, 1005.1971.

[15]  Zhaosong Lu,et al.  Sequential Convex Programming Methods for A Class of Structured Nonlinear Programming , 2012, ArXiv.

[16]  Guoyin Li,et al.  Global Convergence of Splitting Methods for Nonconvex Composite Optimization , 2014, SIAM J. Optim..

[17]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[18]  Carlo Fischione,et al.  On the Convergence of Alternating Direction Lagrangian Methods for Nonconvex Structured Optimization Problems , 2014, IEEE Transactions on Control of Network Systems.

[19]  Jieping Ye,et al.  HONOR: Hybrid Optimization for NOn-convex Regularized problems , 2015, NIPS.

[20]  Shiqian Ma,et al.  Alternating direction method of multipliers for real and complex polynomial optimization models , 2014 .

[21]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[22]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[23]  Ting Kei Pong,et al.  Alternating Direction Method of Multipliers for Nonconvex Background/Foreground Extraction , 2015 .

[24]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[25]  Zhi-Quan Luo,et al.  Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Jieping Ye,et al.  A General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems , 2013, ICML.

[27]  Tong Zhang,et al.  Adaptive Stochastic Alternating Direction Method of Multipliers , 2015, ICML.

[28]  Junfeng Yang,et al.  Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization , 2012, Math. Comput..