A Greedy Homotopy Method for Regression with Nonconvex Constraints

The goal of this paper is to estimate sparse linear regression models, where for a given partition G of input variables, the selected variables are chosen from a diverse set of groups in G. We consider a novel class of nonconvex constraint functions, and develop RepLasso, a greedy homotopy method that exploits geometrical properties of the constraint functions to build a sequence of suitably adapted convex surrogate problems. We prove that in some situations RepLasso recovers the global minima path of the nonconvex problem. Moreover, even if it does not recover the global minima, we prove that it will often do no worse than the Lasso in terms of (signed) support recovery, while in practice outperforming it. We show empirically that the strategy can also be used to improve over various other Lasso-style algorithms. Finally, a GWAS of ankylosing spondylitis highlights our method’s practical utility.

[1]  Francis R. Bach,et al.  Bolasso: model consistent Lasso estimation through the bootstrap , 2008, ICML '08.

[2]  Peng Zhao,et al.  On Model Selection Consistency of Lasso , 2006, J. Mach. Learn. Res..

[3]  Noah Simon,et al.  A Sparse-Group Lasso , 2013 .

[4]  Francis R. Bach,et al.  Structured Variable Selection with Sparsity-Inducing Norms , 2009, J. Mach. Learn. Res..

[5]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[6]  D. Hunter,et al.  Variable Selection using MM Algorithms. , 2005, Annals of statistics.

[7]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[8]  S. Rosset,et al.  Piecewise linear regularized solution paths , 2007, 0708.2197.

[9]  Po-Ling Loh,et al.  Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima , 2013, J. Mach. Learn. Res..

[10]  Bruno Torrésani,et al.  Sparsity and persistence: mixed norms provide simple signal models with dependent coefficients , 2009, Signal Image Video Process..

[11]  Shuiwang Ji,et al.  SLEP: Sparse Learning with Efficient Projections , 2011 .

[12]  Zhaoran Wang,et al.  OPTIMAL COMPUTATIONAL AND STATISTICAL RATES OF CONVERGENCE FOR SPARSE NONCONVEX LEARNING PROBLEMS. , 2013, Annals of statistics.

[13]  Martin J. Wainwright,et al.  Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$ -Constrained Quadratic Programming (Lasso) , 2009, IEEE Transactions on Information Theory.

[14]  Nebojsa Jojic,et al.  Variable Selection through Correlation Sifting , 2011, RECOMB.

[15]  Rong Jin,et al.  Exclusive Lasso for Multi-task Feature Selection , 2010, AISTATS.

[16]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[17]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[18]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[19]  Robert Tibshirani,et al.  The Entire Regularization Path for the Support Vector Machine , 2004, J. Mach. Learn. Res..

[20]  Simon C. Potter,et al.  Association scan of 14,500 nonsynonymous SNPs in four diseases identifies autoimmunity variants , 2007, Nature Genetics.

[21]  R. Tibshirani,et al.  "Preconditioning" for feature selection and regression in high-dimensional problems , 2007, math/0703858.

[22]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[23]  Paul Weston,et al.  Interaction between ERAP1 and HLA-B27 in ankylosing spondylitis implicates peptide handling in the mechanism for HLA-B27 in disease susceptibility , 2011, Nature Genetics.

[24]  Honglak Lee,et al.  Efficient L1 Regularized Logistic Regression , 2006, AAAI.

[25]  Julien Mairal,et al.  Complexity Analysis of the Lasso Regularization Path , 2012, ICML.

[26]  Jinzhu Jia,et al.  Preconditioning to comply with the Irrepresentable Condition , 2012, 1208.5584.

[27]  Hongzhe Li,et al.  Group SCAD regression analysis for microarray time course gene expression data , 2007, Bioinform..

[28]  R. Tibshirani.  The Lasso Problem and Uniqueness , 2012, 1206.0313.

[29]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[30]  Nebojsa Jojic,et al.  A Comparative Framework for Preconditioned Lasso Algorithms , 2013, NIPS.

[31]  Lin Xiao,et al.  A Proximal-Gradient Homotopy Method for the L1-Regularized Least-Squares Problem , 2012, ICML.

[32]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[33]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[34]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .