A Model Free Perspective for Linear Regression: Uniform-in-model Bounds for Post Selection Inference

For the last two decades, high-dimensional data and methods have proliferated throughout the literature. The classical technique of linear regression, however, has not lost its touch in applications. Most high-dimensional estimation techniques can be seen as variable selection tools which lead to a smaller set of variables where classical linear regression technique applies. In this paper, we prove estimation error and linear representation bounds for the linear regression estimator uniformly over (many) subsets of variables. Based on deterministic inequalities, our results provide "good" rates when applied to both independent and dependent data. These results are useful in correctly interpreting the linear regression estimator obtained after exploring the data and also in post model-selection inference. All the results are derived under no model assumptions and are non-asymptotic in nature.

[1]  Ing Rj Ser.  Approximation Theorems of Mathematical Statistics , 1980 .

[2]  W. V. Zwet,et al.  A Berry-Esseen bound for symmetric statistics , 1984 .

[3]  S. Portnoy.  Asymptotic Behavior of $M$-Estimators of $p$ Regression Parameters when $p^2/n$ is Large. I. Consistency , 1984 .

[4]  S. Portnoy.  Asymptotic behavior of M-estimators of p regression parameters when p , 1985 .

[5]  S. Portnoy.  Asymptotic Behavior of Likelihood Methods for Exponential Families when the Number of Parameters Tends to Infinity , 1988 .

[6]  D. Pollard.  Empirical Processes: Theory and Applications , 1990 .

[7]  Q. Shao,et al.  A general bahadur representation of M-estimators and its application to linear regression with nonstochastic designs , 1996 .

[8]  B. M. Pötscher,et al.  Dynamic Nonlinear Econometric Models , 1997 .

[9]  Joseph P. Romano,et al.  A more general central limit theorem for m-dependent random variables with unbounded m , 2000 .

[10]  W. Wu,et al.  Nonlinear system theory: another look at dependence. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[11]  E. Leifer,et al.  Adaptive Regression , 2005, Journal of biopharmaceutical statistics.

[12]  Convergence of the optimal M-estimator over a parametric family of M-estimators , 2005 .

[13]  R. Adamczak.  A tail inequality for suprema of unbounded empirical processes with applications to Markov chains , 2007, 0709.3110.

[14]  Weidong Liu,et al.  ASYMPTOTICS OF SPECTRAL DENSITY ESTIMATES , 2009, Econometric Theory.

[15]  E. Rio.  Moment Inequalities for Sums of Dependent Random Variables under Projective Conditions , 2009 .

[16]  A. Belloni,et al.  Least Squares After Model Selection in High-Dimensional Sparse Models , 2009, 1001.0188.

[17]  J. Mielniczuk,et al.  A new look at measuring dependence , 2010 .

[18]  R. Vershynin.  How Close is the Sample Covariance Matrix to the Actual Covariance Matrix? , 2010, 1004.3484.

[19]  O. Catoni.  Challenging the empirical mean and empirical variance: a deviation study , 2010, 1009.2048.

[20]  Martin J. Wainwright,et al.  Minimax Rates of Estimation for High-Dimensional Linear Regression Over $\ell_q$ -Balls , 2009, IEEE Transactions on Information Theory.

[21]  Po-Ling Loh,et al.  High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity , 2011, NIPS.

[22]  Nathan Srebro,et al.  Fast-rate and optimistic-rate error bounds for L1-regularized regression , 2011, 1108.0373.

[23]  Yaniv Plan,et al.  One‐Bit Compressed Sensing by Linear Programming , 2011, ArXiv.

[24]  M. Yuan,et al.  Adaptive covariance matrix estimation through block thresholding , 2012, 1211.0459.

[25]  Weidong Liu,et al.  Probability and moment inequalities under dependence , 2013 .

[26]  A. Buja,et al.  Valid post-selection inference , 2013, 1306.1059.

[27]  Shie Mannor,et al.  Robust Sparse Regression under Adversarial Corruption , 2013, ICML.

[28]  A. Buja,et al.  Models as Approximations, Part I: A Conspiracy of Nonlinearity and Random Regressors in Linear Regression , 2014, 1404.1578.

[29]  Stanislav Minsker.  Geometric median and robust estimation in Banach spaces , 2013, 1308.1334.

[30]  W. Wu,et al.  Gaussian Approximation for High Dimensional Time Series , 2015, 1508.07036.

[31]  Y. Wu,et al.  Performance bounds for parameter estimates of high-dimensional linear models with correlated errors , 2016 .

[32]  Xiaohan Wei,et al.  Estimation of the covariance structure of heavy-tailed distributions , 2017, NIPS.

[33]  O. Catoni,et al.  Dimension-free PAC-Bayesian bounds for matrices, vectors, and linear least squares regression , 2017, 1712.02747.

[34]  Arun K. Kuchibhotla,et al.  Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression , 2018, 1804.02605.

[35]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[36]  Soumendu Sundar Mukherjee,et al.  Weak convergence and empirical processes , 2019 .