Performance of Empirical Risk Minimization for Linear Regression with Dependent Data

This paper establishes bounds on the performance of empirical risk minimization for large-dimensional linear regression. We generalize existing results by allowing the data to be dependent and heavy-tailed. The analysis covers both the cases of identically and heterogeneously distributed observations. Our analysis is nonparametric in the sense that the relationship between the regressand and the regressors is assumed to be unknown. The main results of this paper indicate that the empirical risk minimizer achieves the optimal performance (up to a logarithmic factor) in a dependent data setting.

[1]  Arkadi Nemirovski,et al.  Topics in Non-Parametric Statistics , 2000 .

[2]  G. Lugosi,et al.  Empirical risk minimization for heavy-tailed losses , 2014, 1406.2462.

[3]  S. Kotz,et al.  Symmetric Multivariate and Related Distributions , 1989 .

[4]  S. Mendelson.  Learning without concentration for general loss functions , 2014, 1410.3192.

[5]  P. Phillips,et al.  High-dimensional VARs with common factors , 2022, Journal of Econometrics.

[6]  E. Rio.  The Functional Law of the Iterated Logarithm for Stationary Strongly Mixing Sequences , 1995 .

[7]  John Odenckantz,et al.  Nonparametric Statistics for Stochastic Processes: Estimation and Prediction , 2000, Technometrics.

[8]  Andrii Babii,et al.  High-Dimensional Granger Causality Tests with an Application to VIX and News , 2020, Journal of Financial Econometrics.

[9]  Shahar Mendelson,et al.  Regularization and the small-ball method II: complexity dependent error rates , 2016, J. Mach. Learn. Res..

[10]  D. Andrews.  Asymptotic Normality of Series Estimators for Nonparametric and Semiparametric Regression Models , 1991 .

[11]  Alexei Onatski,et al.  Asymptotics of the principal components estimator of large factor models with weakly influential factors , 2012 .

[12]  M. Hallin,et al.  The Generalized Dynamic-Factor Model: Identification and Estimation , 2000, Review of Economics and Statistics.

[13]  Jianqing Fan,et al.  High Dimensional Covariance Matrix Estimation in Approximate Factor Models , 2011, Annals of statistics.

[14]  AUTOMATED ESTIMATION OF VECTOR ERROR CORRECTION MODELS , 2015, Econometric Theory.

[15]  B. Hansen.  UNIFORM CONVERGENCE RATES FOR KERNEL ESTIMATION WITH DEPENDENT DATA , 2008, Econometric Theory.

[16]  A. Kock,et al.  Oracle Inequalities for High Dimensional Vector Autoregressions , 2012, 1311.0811.

[17]  M. Tanner,et al.  RISK MINIMIZATION FOR TIME SERIES BINARY CHOICE WITH VARIABLE SELECTION , 2010, Econometric Theory.

[18]  G. Kapetanios,et al.  ESTIMATION OF TIME-VARYING COVARIANCE MATRICES FOR LARGE DATASETS , 2018, Econometric Theory.

[19]  Alexandre B. Tsybakov,et al.  Optimal Rates of Aggregation , 2003, COLT.

[20]  Pentti Saikkonen,et al.  ERGODICITY, MIXING, AND EXISTENCE OF MOMENTS OF A CLASS OF MARKOV MODELS WITH APPLICATIONS TO GARCH AND ACD MODELS , 2008, Econometric Theory.

[21]  C. J. Stone,et al.  Additive Regression and Other Nonparametric Models , 1985 .

[22]  I. Ibragimov,et al.  Some Limit Theorems for Stationary Processes , 1962 .

[23]  K. Knight,et al.  An alternative to unit root tests: Bridge estimators differentiate between nonstationary versus stationary models and select optimal lag , 2013 .

[24]  Jean-Yves Audibert,et al.  Robust linear least squares regression , 2010, 1010.0074.

[25]  Eduardo F. Mendes,et al.  ℓ1-regularization of high-dimensional time-series models with non-Gaussian and heteroskedastic errors , 2016 .

[26]  P. Massart,et al.  Minimum contrast estimators on sieves: exponential bounds and rates of convergence , 1998 .

[27]  W. Newey,et al.  Convergence rates and asymptotic normality for series estimators , 1997 .

[28]  Denis Bosq,et al.  Nonparametric statistics for stochastic processes , 1996 .

[29]  Eckhard Liebscher,et al.  Strong convergence of sums of α-mixing random variables with applications to density estimation , 1996 .

[30]  J. Stock,et al.  Forecasting Using Principal Components From a Large Number of Predictors , 2002 .

[31]  Yingcun Xia,et al.  Semiparametric Regression Models , 2011, International Encyclopedia of Statistical Science.

[32]  A. Tsybakov,et al.  Aggregation for Gaussian regression , 2007, 0710.3654.

[33]  H. White.  Asymptotic theory for econometricians , 1985 .

[34]  Shahar Mendelson,et al.  Learning without Concentration , 2014, COLT.

[35]  Xiaohong Chen.  Chapter 76 Large Sample Sieve Estimation of Semi-Nonparametric Models , 2007 .

[36]  J. Bai,et al.  Determining the Number of Factors in Approximate Factor Models , 2000 .

[37]  S. Mendelson,et al.  Performance of empirical risk minimization in linear aggregation , 2014, 1402.5763.

[38]  Dennis Kristensen,et al.  UNIFORM CONVERGENCE RATES OF KERNEL ESTIMATORS WITH HETEROGENEOUS DEPENDENT DATA , 2009, Econometric Theory.

[39]  S. Mendelson,et al.  Regularization and the small-ball method I: sparse recovery , 2016, 1601.05584.