Models as Approximations, Part I: A Conspiracy of Nonlinearity and Random Regressors in Linear Regression

More than thirty years ago Halbert White inaugurated a "model-robust" form of statistical inference based on the "sandwich estimator" of standard error. This estimator is known to be "heteroskedasticity-consistent", but it is less well-known to be "nonlinearity-consistent" as well. Nonlinearity, however, raises fundamental issues because regressors are no longer ancillary, hence can't be treated as fixed. The consequences are severe: (1)~the regressor distribution affects the slope parameters, and (2)~randomness of the regressors conspires with the nonlinearity to create sampling variability in slope estimates --- even in the complete absence of error. For these observations to make sense it is necessary to re-interpret population slopes and view them not as parameters in a generative model but as statistical functionals associated with OLS fitting as it applies to largely arbitrary joint $\xy$~distributions. In such a "model-robust" approach to linear regression, the meaning of slope parameters needs to be rethought and inference needs to be based on model-robust standard errors that can be estimated with sandwich plug-in estimators or with the $\xy$~bootstrap. Theoretically, model-robust and model-trusting standard errors can deviate by arbitrary magnitudes either way. In practice, a diagnostic test can be used to detect significant deviations on a per-slope basis.

[1]  A. Buja,et al.  Calibrated Percentile Double Bootstrap For Robust Linear Regression Inference , 2015, 1511.00273.

[2]  Sara van de Geer,et al.  High-dimensional inference in misspecified linear models , 2015, 1503.06426.

[3]  D. Donoho,et al.  Variance Breakdown of Huber (M)-estimators: $n/p \rightarrow m \in (1,\infty)$ , 2015, 1503.02106.

[4]  Po-Ling Loh,et al.  Statistical consistency and asymptotic normality for high-dimensional robust M-estimators , 2015, ArXiv.

[5]  Dennis L. Sun,et al.  Exact post-selection inference, with application to the lasso , 2013, 1311.6238.

[6]  P. Bickel,et al.  Optimal M-estimation in high-dimensional regression , 2013, Proceedings of the National Academy of Sciences.

[7]  A. Buja,et al.  Valid post-selection inference , 2013, 1306.1059.

[8]  Thomas Lumley,et al.  Model-Robust Regression and a Bayesian `Sandwich' Estimator , 2010, 1101.1402.

[9]  Donald Ylvisaker,et al.  Counting the Homeless in Los Angeles County , 2008, 0805.2840.

[10]  A. Gelman,et al.  Splitting a Predictor at the Upper Quarter or Third and the Lower Quarter or Third , 2007 .

[11]  D. Freedman,et al.  On The So-Called “Huber Sandwich Estimator” and “Robust Standard Errors” , 2006 .

[12]  J. Aldrich.  Fisher and Regression , 2005 .

[13]  E. Mammen.  Empirical process of residuals for high-dimensional linear models , 1996 .

[14]  C. Clogg,et al.  Statistical Methods for Comparing Regression Coefficients Between Models , 1995, American Journal of Sociology.

[15]  Paul D. Allison,et al.  The Impact of Random Predictors on Comparisons of Coefficients Between Models: Comment on Clogg, Petkova, and Haritou , 1995, American Journal of Sociology.

[16]  E. Mammen.  Bootstrap and Wild Bootstrap for High Dimensional Linear Models , 1993 .

[17]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[18]  M. Berman.  A theorem of Jacobi and its generalization , 1988 .

[19]  Changbao Wu,et al.  Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis , 1986 .

[20]  T. Hastie,et al.  Generalized Additive Models , 1986 .

[21]  S. Zeger,et al.  Longitudinal data analysis using generalized linear models , 1986 .

[22]  H. White,et al.  Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties☆ , 1985 .

[23]  B. Efron,et al.  The Jackknife: The Bootstrap and Other Resampling Plans. , 1983 .

[24]  L. Hansen.  Large Sample Properties of Generalized Method of Moments Estimators , 1982 .

[25]  J. Kent.  Robust properties of likelihood ratio tests , 1982 .

[26]  D. Freedman.  Bootstrapping Regression Models , 1981 .

[27]  H. White.  A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity , 1980 .

[28]  H. White.  Using Least Squares to Approximate Unknown Regression Functions , 1980 .

[29]  R. Berk,et al.  CONSISTENCY A POSTERIORI , 1970 .

[30]  R. Berk,et al.  Limiting Behavior of Posterior Distributions when the Model is Incorrect , 1966 .

[31]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[32]  F. Eicker.  Asymptotic Normality and Consistency of the Least Squares Estimators for Families of Linear Regressions , 1963 .

[33]  Laurie Davies,et al.  Data Analysis and Approximate Models , 2015 .

[34]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[35]  F. Götze,et al.  RESAMPLING FEWER THAN n OBSERVATIONS: GAINS, LOSSES, AND REMEDIES FOR LOSSES , 2012 .

[36]  L. Wasserman.  Low Assumptions, High Dimensions , 2011 .

[37]  P. Hall.  The Bootstrap and Edgeworth Expansion , 1992 .

[38]  N. Weber,et al.  The jackknife and heteroskedasticity: Consistent variance estimation for regression models , 1986 .

[39]  J. A. Hartigan,et al.  Asymptotic Normality of Posterior Distributions , 1983 .

[40]  G. Box.  Robustness in the Strategy of Scientific Model Building. , 1979 .

[41]  P. J. Huber.  The behavior of maximum likelihood estimates under nonstandard conditions , 1967 .