Computation of second-order directional stationary points for group sparse optimization

ABSTRACT We consider a nonconvex and nonsmooth group sparse optimization problem where the penalty function is the sum of compositions of a folded concave function and the vector norm for each group variable. We show that under some mild conditions a first-order directional stationary point is a strict local minimizer that fulfils the first-order growth condition, and a second-order directional stationary point is a strong local minimizer that fulfils the second-order growth condition. In order to compute second-order directional stationary points, we construct a twice continuously differentiable smoothing problem and show that any accumulation point of the sequence of second-order stationary points of the smoothing problem is a second-order directional stationary point of the original problem. We give numerical examples to illustrate how to compute a second-order directional stationary point by the smoothing method.

[1]  Jian Huang,et al.  Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors , 2012, Statistics and Computing.

[2]  Jean-Philippe Vert,et al.  Group lasso with overlap and graph lasso , 2009, ICML '09.

[4]  Jack Xin,et al.  Difference-of-Convex Learning: Directional Stationarity, Optimality, and Sparsity , 2017, SIAM J. Optim..

[5]  Wenjiang J. Fu,et al.  Asymptotics for lasso-type estimators , 2000 .

[6]  Mingyi Hong,et al.  Local Minimizers and Second-Order Conditions in Composite Piecewise Programming via Directional Derivatives , 2017 .

[7]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[8]  Xiaojun Chen,et al.  Optimality Conditions and a Smoothing Trust Region Newton Method for NonLipschitz Optimization , 2013, SIAM J. Optim..

[9]  Mila Nikolova,et al.  Efficient Reconstruction of Piecewise Constant Images Using Nonsmooth Nonconvex Minimization , 2008, SIAM J. Imaging Sci..

[10]  Xiaojun Chen,et al.  Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization , 2015, Math. Program..

[11]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[12]  H. Zou,et al.  STRONG ORACLE OPTIMALITY OF FOLDED CONCAVE PENALIZED ESTIMATION. , 2012, Annals of statistics.

[13]  P. Zhao,et al.  The composite absolute penalties family for grouped and hierarchical variable selection , 2009, 0909.0411.

[14]  A. Ben-Tal,et al.  Necessary and sufficient optimality conditions for a class of nonsmooth minimization problems , 1982, Math. Program..

[15]  P. Bühlmann,et al.  The group lasso for logistic regression , 2008 .

[16]  Tong Zhang,et al.  Analysis of Multi-stage Convex Relaxation for Sparse Regularization , 2010, J. Mach. Learn. Res..

[17]  Yaoliang Yu,et al.  Generalized Conditional Gradient for Sparse Estimation , 2014, J. Mach. Learn. Res..

[18]  Yongdai Kim,et al.  Sparse optimization for nonconvex group penalized estimation , 2016 .

[19]  Xiaoqi Yang,et al.  On second-order directional derivatives , 1996 .

[20]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[21]  Bangti Jin,et al.  Group Sparse Recovery via the $\ell ^0(\ell ^2)$ Penalty: Theory and Algorithm , 2016, IEEE Transactions on Signal Processing.

[22]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[23]  Junzhou Huang,et al.  The Benefit of Group Sparsity , 2009 .

[24]  Cun-Hui Zhang,et al.  A group bridge approach for variable selection , 2009, Biometrika.

[25]  Runze Li,et al.  Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery , 2006, math/0602133.

[26]  Tao Yao,et al.  Folded concave penalized sparse linear regression: sparsity, statistical performance, and algorithmic theory for local solutions , 2017, Math. Program..

[27]  Xiaojun Chen,et al.  Optimality and Complexity for Constrained Optimization Problems with Nonconvex Regularization , 2017, Math. Oper. Res..

[28]  Xiaojun Chen,et al.  Smoothing methods for nonsmooth, nonconvex minimization , 2012, Math. Program..

[29]  Bangti Jin,et al.  Group Sparse Recovery via the ℓ0(ℓ2) Penalty: Theory and Algorithm , 2016, ArXiv.

[30]  Le Thi Hoai An,et al.  DC approximation approaches for sparse optimization , 2014, Eur. J. Oper. Res..

[31]  Eunho Yang,et al.  Sparse + Group-Sparse Dirty Models: Statistical Guarantees without Unreasonable Conditions and a Case for Non-Convexity , 2017, ICML.

[32]  M. Vidyasagar,et al.  Error Bounds for Compressed Sensing Algorithms With Group Sparsity: A Unified Approach , 2015, 1512.08673.

[33]  Yi Yang,et al.  A fast unified algorithm for solving group-lasso penalize learning problems , 2014, Statistics and Computing.

[34]  Fengrong Wei,et al.  Group coordinate descent algorithms for nonconvex penalized regression , 2012, Comput. Stat. Data Anal..

[35]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[36]  Y. Ye,et al.  Lower Bound Theory of Nonzero Entries in Solutions of ℓ2-ℓp Minimization , 2010, SIAM J. Sci. Comput..

[37]  W. Marsden.  I and J , 2012 .

[38]  Lichong,et al.  Group sparse optimization via lp,q regularization , 2017 .

[39]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[40]  Ya-Xiang Yuan,et al.  Conditions for convergence of trust region algorithms for nonsmooth optimization , 1985, Math. Program..

[41]  Jong-Shi Pang,et al.  Computing B-Stationary Points of Nonsmooth DC Programs , 2015, Math. Oper. Res..