Covariance Matrix Estimation in Time Series

Abstract Covariances play a fundamental role in the theory of time series, and they are critical quantities that are needed in both spectral and time domain analysis. Estimation of covariance matrices is needed in the construction of confidence regions for unknown parameters, hypothesis testing, principal component analysis, prediction, discriminant analysis, among others. In this chapter, we consider both low and high-dimensional covariance matrix estimation problems and present a review for asymptotic properties of sample covariances and covariance matrix estimates. In particular, we shall provide an asymptotic theory for estimates of high-dimensional covariance matrices in time series and a consistency result for covariance matrix estimates for estimated parameters.

[1]  Qiwei Yao,et al.  Inference in ARCH and GARCH models with heavy-tailed errors , 2003 .

[2]  Jianhua Z. Huang,et al.  Covariance matrix selection and estimation via penalised normal likelihood , 2006 .

[3]  D. Luenberger,et al.  Estimation of structured covariance matrices , 1982, Proceedings of the IEEE.

[4]  M. Pourahmadi.  Covariance Estimation: The GLM and Regularization Perspectives , 2011, 1202.1661.

[5]  Björn E. Ottersten,et al.  Structured covariance matrix estimation: a parametric approach , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[6]  Han Xiao,et al.  Covariance matrix estimation for stationary time series , 2011, 1105.4563.

[7]  W. Wu,et al.  On linear processes with dependent innovations , 2005 .

[8]  E. J. Hannan,et al.  The Asymptotic Distribution of Serial Covariances , 1976 .

[9]  C. Heyde.  Quasi-likelihood and its application : a general approach to optimal parameter estimation , 1998 .

[10]  Michael I. Miller,et al.  On the existence of positive-definite maximum-likelihood estimates of structured covariance matrices , 1988, IEEE Trans. Inf. Theory.

[11]  Xinwei Deng,et al.  Large Gaussian Covariance Matrix Estimation With Markov Structures , 2009 .

[12]  N. Bingham.  INDEPENDENT AND STATIONARY SEQUENCES OF RANDOM VARIABLES , 1973 .

[13]  D. Paul,et al.  Asymptotics of the leading sample eigenvalues for a spiked covariance model , 2004 .

[14]  Paul I. Nelson,et al.  On Conditional Least Squares Estimation for Stochastic Processes , 1978 .

[15]  Frank Dietrich,et al.  Robust Signal Processing for Wireless Communications , 2007 .

[16]  Gilbert MacKenzie,et al.  On modelling mean‐covariance structures in longitudinal studies , 2003 .

[17]  Han Xiao,et al.  Asymptotic Inference of Autocovariances of Stationary Processes , 2011, 1105.3423.

[18]  D. B. Preston.  Spectral Analysis and Time Series , 1983 .

[19]  Ernst Eberlein,et al.  Dependence in probability and statistics : a survey of recent results (Oberwolfach, 1985) , 1988 .

[20]  Otto Toeplitz,et al.  Zur Theorie der quadratischen und bilinearen Formen von unendlichvielen Veränderlichen , 1911 .

[21]  Jianqing Fan,et al.  High dimensional covariance matrix estimation using a factor model , 2007, math/0701124.

[22]  Victor Solo,et al.  Asymptotics for Linear Processes , 1992 .

[23]  D. Zimmerman,et al.  Antedependence Models for Longitudinal Data , 2009 .

[24]  Adam J. Rothman,et al.  A new approach to Cholesky-based covariance regularization in high dimensions , 2009, 0903.0645.

[25]  Wei Biao Wu,et al.  M-estimation of linear models with dependent errors , 2004, math/0412268.

[26]  R. Dahlhaus.  Fitting time series models to nonstationary processes , 1997 .

[27]  Z. Bai.  METHODOLOGIES IN SPECTRAL ANALYSIS OF LARGE DIMENSIONAL RANDOM MATRICES , A REVIEW , 1999 .

[28]  W. Wu,et al.  Covariances Estimation for Long-Memory Processes , 2010, Advances in Applied Probability.

[29]  A. Zeileis.  Econometric Computing with HC and HAC Covariance Matrix Estimators , 2004 .

[30]  M. Pourahmadi,et al.  BANDING SAMPLE AUTOCOVARIANCE MATRICES OF STATIONARY PROCESSES , 2009 .

[31]  David R. Brillinger,et al.  Time series in the frequency domain , 1985 .

[32]  P. Bickel,et al.  Covariance regularization by thresholding , 2009, 0901.3079.

[33]  Tom Leonard,et al.  The Matrix-Logarithmic Covariance Model , 1996 .

[34]  Amir Dembo,et al.  The relation between maximum likelihood estimation of structured covariance matrices and periodograms , 1986, IEEE Trans. Acoust. Speech Signal Process..

[35]  K. Wachter.  The Strong Limits of Random Matrix Spectra for Sample Matrices of Independent Elements , 1978 .

[36]  Patrick J. Heagerty,et al.  Window Subsampling of Estimating Functions with Application to Regression Models , 2000 .

[37]  R. R. Bahadur.  A Note on Quantiles in Large Samples , 1966 .

[38]  Patrick J. Heagerty,et al.  Weighted empirical adaptive variance estimators for correlated data regression , 1999 .

[39]  F. Eicker.  Asymptotic Normality and Consistency of the Least Squares Estimators for Families of Linear Regressions , 1963 .

[40]  H. White,et al.  Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties☆ , 1985 .

[41]  W. Wu.  An asymptotic theory for sample covariances of Bernoulli shifts , 2009 .

[42]  P. Bickel,et al.  Banded regularization of autocovariance matrices in application to parameter estimation and forecasting of time series , 2011 .

[43]  H. White.  A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity , 1980 .

[44]  M. Pourahmadi.  Joint mean-covariance models with applications to longitudinal data: Unconstrained parameterisation , 1999 .

[45]  U. Grenander,et al.  Statistical Spectral Analysis of Time Series Arising from Stationary Stochastic Processes , 1953 .

[46]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[47]  W. Wu,et al.  Asymptotic theory for stationary processes , 2011 .

[48]  U. Grenander,et al.  Toeplitz Forms And Their Applications , 1958 .

[49]  Adam J. Rothman,et al.  Generalized Thresholding of Large Covariance Matrices , 2009 .

[50]  Donald W. K. Andrews,et al.  An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator , 1992 .

[51]  V. Marčenko,et al.  DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .

[52]  Noureddine El Karoui.  Spectrum estimation for large dimensional covariance matrices using random matrix theory , 2006, math/0609418.

[53]  Noureddine El Karoui.  Tracy–Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices , 2005, math/0503109.

[54]  V. Borkar.  White-noise representations in stochastic realization theory , 1993 .

[55]  M. Yuan,et al.  Model selection and estimation in the Gaussian graphical model , 2007 .

[56]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[57]  Weidong Liu,et al.  ASYMPTOTICS OF SPECTRAL DENSITY ESTIMATES , 2009, Econometric Theory.

[58]  Bruce E. Hansen,et al.  Consistent Covariance Matrix Estimation for Dependent Heterogeneous Processes , 1992 .

[59]  Mohsen Pourahmadi,et al.  Foundations of Time Series Analysis and Prediction Theory , 2001 .

[60]  M. Rosenblatt.  A comment on a conjecture of N. Wiener , 2009 .

[61]  J. Davidson,et al.  Consistency of Kernel Estimators of Heteroscedastic and Autocorrelated Covariance Matrices , 2000 .

[62]  D. Ornstein,et al.  An example of a Kolmogorov automorphism that is not a Bernoulli shift , 1973 .

[63]  P. Doukhan.  Mixing: Properties and Examples , 1994 .

[64]  M. Rosenblatt.  Stationary sequences and random fields , 1985 .

[65]  Richard C. Bradley,et al.  Introduction to strong mixing conditions , 2007 .

[66]  D. Politis,et al.  Banded and tapered estimates for autocovariance matrices and the linear process bootstrap , 2010 .

[67]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[68]  N. Wiener,et al.  Nonlinear Problems in Random Theory , 1964 .

[69]  T. Cipra.  Statistical Analysis of Time Series , 2010 .

[70]  M. Bartlett.  On the Theoretical Specification and Sampling Properties of Autocorrelated Time‐Series , 1946 .

[71]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[72]  S. Geman.  A Limit Theorem for the Norm of Random Matrices , 1980 .

[73]  K. Worsley,et al.  Random fields of multivariate test statistics, with applications to shape analysis , 2008, 0803.1708.

[74]  A. Tanikawa.  Martingale limit theorem and its application to an ergodic controlled Markov chain , 1993 .

[75]  P. Bickel,et al.  Regularized estimation of large covariance matrices , 2008, 0803.1909.

[76]  Clifford S. Stein.  Estimation of a covariance matrix , 1975 .

[77]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series , 1964 .

[78]  I. Ibragimov,et al.  Independent and stationary sequences of random variables , 1971 .

[79]  A Nguyen.  On the uniqueness of the maximum-likeliwood estimate of structured covariance matrices , 1984 .

[80]  Harrison H. Zhou,et al.  Optimal rates of convergence for covariance matrix estimation , 2010, 1010.3866.

[81]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[82]  J. W. Silverstein,et al.  Spectral Analysis of Large Dimensional Random Matrices , 2009 .

[83]  A CENTRAL LIMIT THEOREM FOR m(n) AUTOCOVARIANCES , 1997 .

[84]  SOME LIMIT THEORY FOR AUTOCOVARIANCES WHOSE ORDER DEPENDS ON SAMPLE SIZE , 2003, Econometric Theory.

[85]  S. Kalikow,et al.  T, T-1 transformation is not loosely Bernoulli* , 1982 .

[86]  Jonathan R. M. Hosking,et al.  Asymptotic distributions of the sample mean, autocovariances, and autocorrelations of long-memory time series , 1996 .

[87]  W. Wu,et al.  Nonlinear system theory: another look at dependence. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[88]  M. Pourahmadi,et al.  Nonparametric estimation of large covariance matrices of longitudinal data , 2003 .

[89]  E. J. Hannan,et al.  Multiple time series , 1970 .

[90]  W. Newey,et al.  Automatic Lag Selection in Covariance Matrix Estimation , 1994 .

[91]  Jianqing Fan,et al.  Sparsistency and Rates of Convergence in Large Covariance Matrix Estimation. , 2007, Annals of statistics.

[92]  C. Tracy,et al.  Introduction to Random Matrices , 1992, hep-th/9210073.

[93]  A CLT for regularized sample covariance matrices , 2006, math/0612791.

[94]  D. Andrews.  Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation , 1991 .

[95]  Lars P. B. Christensen.  An EM-Algorithm for Band-Toeplitz Covariance Matrix Estimation , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[96]  Q. Shao,et al.  A general bahadur representation of M-estimators and its application to linear regression with nonstochastic designs , 1996 .