Consistent Robust Regression

We present the first efficient and provably consistent estimator for the robust regression problem. The area of robust learning and optimization has generated a significant amount of interest in the learning and statistics communities in recent years owing to its applicability in scenarios with corrupted data, as well as in handling model mis-specifications. In particular, special interest has been devoted to the fundamental problem of robust linear regression where estimators that can tolerate corruption in up to a constant fraction of the response variables are widely studied. Surprisingly however, to this date, we are not aware of a polynomial time estimator that offers a consistent estimate in the presence of dense, unbounded corruptions. In this work we present such an estimator, called CRR. This solves an open problem put forward in the work of (Bhatia et al, 2015). Our consistency analysis requires a novel two-stage proof technique involving a careful analysis of the stability of ordered lists which may be of independent interest. We show that CRR not only offers consistent estimates, but is empirically far superior to several other recently proposed algorithms for the robust regression problem, including extended Lasso and the TORRENT algorithm. In comparison, CRR offers comparable or better model recovery but with runtimes that are faster by an order of magnitude.

[1]  P. Rousseeuw.  Least Median of Squares Regression , 1984 .

[2]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[3]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[4]  John Wright,et al.  Dense error correction via l1-minimization , 2008, ICASSP.

[5]  Rahul Garg,et al.  Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry property , 2009, ICML '09.

[6]  Richard G. Baraniuk,et al.  A simple proof that random matrices are democratic , 2009, ArXiv.

[7]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[9]  Helmut Bölcskei,et al.  Recovery of Sparsely Corrupted Signals , 2011, IEEE Transactions on Information Theory.

[10]  Yin Chen,et al.  Fused sparsity and robust estimation for linear models with unknown variance , 2012, NIPS.

[11]  Trac D. Tran,et al.  Robust Lasso With Missing and Grossly Corrupted Observations , 2011, IEEE Transactions on Information Theory.

[12]  Shie Mannor,et al.  Robust Sparse Regression under Adversarial Corruption , 2013, ICML.

[13]  Trac D. Tran,et al.  Exact Recoverability From Dense Corrupted Observations via $\ell _{1}$-Minimization , 2011, IEEE Transactions on Information Theory.

[14]  Allen Y. Yang,et al.  Fast L1-Minimization Algorithms For Robust Face Recognition , 2010, 1007.3753.

[15]  Shie Mannor,et al.  Robust Logistic Regression and Classification , 2014, NIPS.

[16]  Joachim M. Buhmann,et al.  Fast and Robust Least Squares Estimation in Corrupted Linear Models , 2014, NIPS.

[17]  Prateek Jain,et al.  On Iterative Hard Thresholding Methods for High-dimensional M-Estimation , 2014, NIPS.

[18]  Prateek Jain,et al.  Robust Regression via Hard Thresholding , 2015, NIPS.

[19]  Gregory Valiant,et al.  Learning from untrusted data , 2016, STOC.

[20]  Prateek Jain,et al.  Nearly Optimal Robust Matrix Completion , 2016, ICML.