Variable Selection is Hard

Variable selection for sparse linear regression is the problem of finding, given an m p matrix B and a target vector y, a sparse vector x such that Bx approximately equals y. Assuming a standard complexity hypothesis, we show that no polynomial-time algorithm can find ak 0 -sparse x withkBx yk 2 h(m;p), wherek 0 = k 2 log 1 p andh(m;p) = p C1 m 1 C2 , where > 0;C1 > 0;C2 > 0 are arbitrary. This is true even under the promise that there is an unknownk-sparse vector x satisfyingBx = y. We prove a similar result for a statistical version of the problem in which the data are corrupted by noise. To the authors’ knowledge, these are the first hardness results for sparse regression that apply when the algorithm simultaneously hask 0 > k andh(m;p) > 0.

[1]  S. Montgomery-Smith.  The distribution of Rademacher sums , 1990 .

[2]  Sanjeev Arora,et al.  The Hardness of Approximate Optimia in Lattices, Codes, and Systems of Linear Equations , 1993, IEEE Annual Symposium on Foundations of Computer Science.

[3]  Carsten Lund,et al.  Efficient probabilistically checkable proofs and applications to approximations , 1993, STOC.

[4]  Carsten Lund,et al.  On the hardness of approximating minimization problems , 1993, STOC.

[5]  Carsten Lund,et al.  Efficient probabilistic checkable proofs and applications to approximation , 1994, STOC '94.

[6]  Dean P. Foster,et al.  The risk inflation criterion for multiple regression , 1994 .

[7]  Ran Raz,et al.  A parallel repetition theorem , 1995, STOC '95.

[8]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[9]  Dorit S. Hochbaum,et al.  Approximation Algorithms for NP-Hard Problems , 1996 .

[10]  Jacques Stern,et al.  The Hardness of Approximate Optima in Lattices, Codes, and Systems of Linear Equations , 1997, J. Comput. Syst. Sci..

[11]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[12]  U. Feige.  A threshold of ln n for approximating set cover , 1998, JACM.

[13]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[14]  Edoardo Amaldi,et al.  On the Approximability of Minimizing Nonzero Variables or Unsatisfied Relations in Linear Systems , 1998, Theor. Comput. Sci..

[15]  S. Griffis.  EDITOR , 1997, Journal of Navigation.

[16]  Shanmugavelayutham Muthukrishnan,et al.  Data Stream Algorithms , 2005 .

[17]  T. Blumensath,et al.  On the Difference Between Orthogonal Matching Pursuit and Orthogonal Least Squares , 2007 .

[18]  S. Geer,et al.  On the conditions used to prove oracle results for the Lasso , 2009, 0910.0722.

[19]  Dana Moshkovitz,et al.  The Projection Games Conjecture and the NP-Hardness of ln n-Approximating Set-Cover , 2012, Theory Comput..

[20]  Ali Çivril.  A note on the hardness of sparse approximation , 2013, Inf. Process. Lett..

[21]  Martin J. Wainwright,et al.  Lower bounds on the performance of polynomial-time algorithms for sparse linear regression , 2014, COLT.

[22]  David Steurer,et al.  Analytical approach to parallel repetition , 2013, STOC.