A Theory of High-dimensional Sparse Estimation via Non-Convex Regularized Regression

Non-convex regularized regression improves the performance of high-dimensional sparse estimation. Compared with convex regularizers, one of the important improvements is the weaker requirement on design matrix or, rather, weaker estimation condition. Estimation condition is a core issue for high-dimensional sparse estimation. However, previous works demanded the same estimation conditions as the convex regularized regression, which cannot explain the superiority of non-convex regularizers from the view of estimation condition and limits the further applications of them. This paper fills the gap between theory and experience by proposing new sparse eigenvalue based estimation conditions. For a general family of regularizers, named \xi-sharp concave regularizers, our conditions are weaker than that the convex regularizers need. Moreover, consistent sparse estimations are available not only for the global solutions of regularized regression, but also for the so-called approximate global and approximate stationary (AGAS) solutions. Our results on AGAS solutions are useful for application since we show the robustness of the non-convex regularized regression to the inaccuracy of the solutions and give a theoretical guarantee for the numerical solutions. Also, we give a quality guarantee for any solution that is regarded as an approximate global solution and prove that the desired approximate stationary solutions can be obtained simply by coordinate descent methods. This paper provides a general theory to non-convex high-dimensional sparse estimation and can serve as a guideline for selecting regularizers and developing algorithms for non-convex regularized regression.