Sampling Requirements for Stable Autoregressive Estimation

We consider the problem of estimating the parameters of a linear univariate autoregressive (AR) model with sub-Gaussian innovations from a limited sequence of consecutive observations. Assuming that the parameters are compressible, we analyze the performance of the <inline-formula><tex-math notation="LaTeX">$\ell _1$</tex-math> </inline-formula>-regularized least squares as well as a greedy estimator of the parameters and characterize the sampling tradeoffs required for stable recovery in the nonasymptotic regime. In particular, we show that for a fixed sparsity level, stable recovery of AR parameters is possible when the number of samples scale <italic>sublinearly </italic> with the AR order. Our results improve over existing sampling complexity requirements in AR estimation using the Lasso, when the sparsity level scales faster than the square root of the model order. We further derive sufficient conditions on the sparsity level that guarantee the minimax optimality of the <inline-formula> <tex-math notation="LaTeX">$\ell _1$</tex-math></inline-formula>-regularized least squares estimate. Applying these techniques to simulated data as well as real-world datasets from crude oil prices and traffic speed data confirm our predicted theoretical performance gains in terms of estimation accuracy and model selection.

[1]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[2]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[3]  H. Akaike.  Statistical predictor identification , 1970 .

[4]  Donald B. Percival,et al.  Spectral Analysis for Physical Applications , 1993 .

[5]  Wenjiang J. Fu,et al.  Asymptotics for lasso-type estimators , 2000 .

[6]  Jonas Schmitt,et al.  Likelihood Based Inference In Cointegrated Vector Autoregressive Models , 2016 .

[7]  M. Rudelson,et al.  On sparse reconstruction from Fourier and Gaussian measurements , 2008 .

[8]  O. Antoine,et al.  Theory of Error-correcting Codes , 2022 .

[9]  Justin K. Romberg,et al.  Restricted Isometries for Partial Random Circulant Matrices , 2010, ArXiv.

[10]  Herold Dehling,et al.  Empirical Process Techniques for Dependent Data , 2002 .

[11]  Carlos Carmona,et al.  Travel Time Forecasting and Dynamic Origin-Destination Estimation for Freeways Based on Bluetooth Traffic Monitoring , 2010 .

[12]  Michael E. Mann,et al.  Oscillatory Spatiotemporal Signal Detection in Climate Studies: A Multiple-Taper Spectral Domain Approach , 1999 .

[13]  B. Hofmann-Wellenhof,et al.  Introduction to spectral analysis , 1986 .

[14]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[15]  J. Cadzow.  Maximum Entropy Spectral Analysis , 2006 .

[16]  Y. Wu,et al.  Performance bounds for parameter estimates of high-dimensional linear models with correlated errors , 2016 .

[17]  Kareem E. Baddour,et al.  Autoregressive modeling for fading channel simulation , 2005, IEEE Transactions on Wireless Communications.

[18]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[19]  S. Geer.  On Hoeffding's Inequality for Dependent Random Variables , 2002 .

[20]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[21]  Robert D. Nowak,et al.  Toeplitz Compressed Sensing Matrices With Applications to Sparse Channel Estimation , 2010, IEEE Transactions on Information Theory.

[22]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[23]  N. J. A. Sloane,et al.  Lower bounds for constant weight codes , 1980, IEEE Trans. Inf. Theory.

[24]  S. Johansen.  Likelihood-Based Inference in Cointegrated Vector Autoregressive Models , 1996 .

[25]  Po-Ling Loh,et al.  High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity , 2011, NIPS.

[26]  S. R. Jammalamadaka,et al.  Empirical Processes in M-Estimation , 2001 .

[27]  P. Robinson.  Time Series with Long Memory , 2003 .

[28]  John W. Galbraith,et al.  Autoregression-Based Estimators for ARFIMA Models , 2001 .

[29]  Stephen E. Fienberg,et al.  Testing Statistical Hypotheses , 2005 .

[30]  H. Akaike.  Maximum likelihood identification of Gaussian autoregressive moving average models , 1973 .

[31]  Tong Zhang,et al.  Sparse Recovery With Orthogonal Matching Pursuit Under RIP , 2010, IEEE Transactions on Information Theory.

[32]  Ambuj Tewari,et al.  Regularized Estimation in High Dimensional Time Series under Mixing Conditions , 2016, ArXiv.

[33]  Stephen D. Clark,et al.  Traffic Prediction Using Multivariate Nonparametric Regression , 2003 .

[34]  Peng Zhao,et al.  On Model Selection Consistency of Lasso , 2006, J. Mach. Learn. Res..

[35]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[36]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[37]  R. D'Agostino,et al.  Goodness-of-Fit-Techniques , 1987 .

[38]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[39]  Emmanuel J. Candès,et al.  Modern statistical estimation via oracle inequalities , 2006, Acta Numerica.

[40]  Martin J. Wainwright,et al.  Estimation of (near) low-rank matrices with noise and high-dimensional scaling , 2009, ICML.

[41]  H. Akaike.  Fitting autoregressive models for prediction , 1969 .

[42]  Ambuj Tewari,et al.  Lasso Guarantees for Time Series Estimation Under Subgaussian Tails and $ \beta $-Mixing , 2016 .

[43]  Min Wu,et al.  Sufficient conditions for stable recovery of sparse autoregressive models , 2016, 2016 Annual Conference on Information Science and Systems (CISS).

[44]  W. M. Carey,et al.  Digital spectral analysis: with applications , 1986 .

[45]  Ching-Zong Wei,et al.  Order selection for same-realization predictions in autoregressive processes , 2005, math/0602326.

[46]  Yan Sun,et al.  Simultaneous sparse model selection and coefficient estimation for heavy-tailed autoregressive processes , 2011, 1112.2682.

[47]  R. Shibata.  Asymptotically Efficient Selection of the Order of the Model for Estimating Parameters of a Linear Process , 1980 .

[48]  John W. Galbraith,et al.  On some simple, autoregression-based estimation and identification techniques for ARMA models , 1997 .

[49]  Ralph B. D'Agostino,et al.  Goodness-of-Fit-Techniques , 2020 .

[50]  Donald Poskitt,et al.  Autoregressive approximation in nonstandard situations: the fractionally integrated and non-invertible cases , 2007 .

[51]  Martin J. Wainwright,et al.  Restricted Eigenvalue Properties for Correlated Gaussian Designs , 2010, J. Mach. Learn. Res..

[52]  G. Michailidis,et al.  Regularized estimation in sparse high-dimensional time series models , 2013, 1311.4175.

[53]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[54]  Min Wu,et al.  Robust Estimation of Self-Exciting Point Process Models with Application to Neuronal Modeling , 2015, ArXiv.

[55]  Fang Han,et al.  Transition Matrix Estimation in High Dimensional Time Series , 2013, ICML.

[56]  E.J. Candes.  Compressive Sampling , 2022 .

[57]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[58]  A. R. Cook,et al.  ANALYSIS OF FREEWAY TRAFFIC TIME-SERIES DATA BY USING BOX-JENKINS TECHNIQUES , 1979 .

[59]  Kaveh Farokhi Sadabadi.  Vehicular traffic modelling, data assimilation, estimation and short term travel time prediction , 2014 .

[60]  T. W. Anderson.  Goodness-of-fit tests for autoregressive processes , 1997 .

[61]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[62]  Samir A. Ahmed,et al.  APPLICATION OF TIME-SERIES ANALYSIS TECHNIQUES TO FREEWAY INCIDENT DETECTION , 1982 .

[63]  A. Zeevi,et al.  NONASYMPTOTIC BOUNDS FOR AUTOREGRESSIVE TIME SERIES MODELING , 2001 .

[64]  R. Rudzkis.  Large deviations for estimates of spectrum of stationary series , 1978 .

[65]  Min Wu,et al.  Sparse estimation of self-exciting point processes with application to LGN neural modeling , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[66]  Martin J. Wainwright,et al.  Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$ -Constrained Quadratic Programming (Lasso) , 2009, IEEE Transactions on Information Theory.

[67]  U. Grenander,et al.  Toeplitz Forms And Their Applications , 1958 .

[68]  Chih-Ling Tsai,et al.  Regression coefficient and autoregressive order shrinkage and selection via the lasso , 2007 .

[69]  Y. Nardi,et al.  Autoregressive process modeling via the Lasso procedure , 2008, J. Multivar. Anal..