Strong NP-Hardness Result for Regularized $L_q$-Minimization Problems with Concave Penalty Functions

In this note, we consider the regularize $L_q$-minimization problem ($q\ge 1$) with a general penalty function. We show that if the penalty function is concave but not linear in a neighborhood of zero, then the optimization problem is strongly NP-hard. This result answers the complexity of many regularized optimization problems studied in the literature. It implies that it is impossible to have a fully polynomial-time approximation scheme (FPTAS) for a large class of regularization problems unless P = NP.

[1]  J. Friedman,et al.  A Statistical View of Some Chemometrics Regression Tools , 1993 .

[2]  Jianqing Fan,et al.  Regularization of Wavelet Approximations , 2001 .

[3]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[4]  Tong Zhang,et al.  Analysis of Multi-stage Convex Relaxation for Sparse Regularization , 2010, J. Mach. Learn. Res..

[5]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[6]  Martin J. Wainwright,et al.  Lower bounds on the performance of polynomial-time algorithms for sparse linear regression , 2014, COLT.

[7]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[8]  X. Huo,et al.  Complexity of penalized likelihood estimation , 2010 .

[9]  Zizhuo Wang,et al.  Complexity of Unconstrained L2-Lp Minimization , 2011 .

[10]  Jianqing Fan,et al.  A Selective Overview of Variable Selection in High Dimensional Feature Space. , 2009, Statistica Sinica.

[11]  H. Zou,et al.  Nonconcave penalized composite conditional likelihood estimation of sparse Ising models , 2012, 1208.3555.

[12]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[13]  Zhaoran Wang,et al.  OPTIMAL COMPUTATIONAL AND STATISTICAL RATES OF CONVERGENCE FOR SPARSE NONCONVEX LEARNING PROBLEMS. , 2013, Annals of statistics.

[14]  H. Zou,et al.  STRONG ORACLE OPTIMALITY OF FOLDED CONCAVE PENALIZED ESTIMATION. , 2012, Annals of statistics.

[15]  David S. Johnson,et al.  `` Strong '' NP-Completeness Results: Motivation, Examples, and Implications , 1978, JACM.

[16]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[17]  P. McCullagh,et al.  Generalized Linear Models , 1972, Predictive Analytics.

[18]  Dean P. Foster,et al.  Variable Selection is Hard , 2014, COLT.

[19]  Wei Bian,et al.  Optimality Conditions and Complexity for Non-Lipschitz Constrained Optimization Problems , 2014 .

[20]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[21]  T. Hastie,et al.  SparseNet: Coordinate Descent With Nonconvex Penalties , 2011, Journal of the American Statistical Association.

[22]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[23]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[24]  Jinchi Lv,et al.  A unified approach to model selection and sparse recovery using regularized least squares , 2009, 0905.3573.

[25]  Edoardo Amaldi,et al.  On the Approximability of Minimizing Nonzero Variables or Unsatisfied Relations in Linear Systems , 1998, Theor. Comput. Sci..

[26]  Xiaojun Chen,et al.  Complexity of unconstrained $$L_2-L_p$$ minimization , 2011, Math. Program..