Joint Covariance Estimation With Mutual Linear Structure

We consider the problem of joint estimation of structured covariance matrices. Assuming the structure is unknown, estimation is achieved using heterogeneous training sets. Namely, given groups of measurements coming from centered populations with different covariances, our aim is to determine the mutual structure of these covariance matrices and estimate them. Supposing that the covariances span a low dimensional affine subspace in the space of symmetric matrices, we develop a new efficient algorithm discovering the structure and using it to improve the estimation. Our technique is based on the application of principal component analysis in the matrix space. We also derive an upper performance bound of the proposed algorithm in the Gaussian scenario and compare it with the Cramer–Rao lower bound. Numerical simulations are presented to illustrate the performance benefits of the proposed method.

[1]  L. Scharf,et al.  Statistical Signal Processing: Detection, Estimation, and Time Series Analysis , 1991 .

[2]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[3]  Joseph A. O'Sullivan,et al.  The use of maximum likelihood estimation for forming images of diffuse radar targets from delay-Doppler data , 1989, IEEE Trans. Inf. Theory.

[4]  Jean-Yves Tourneret,et al.  Covariance Matrix Estimation With Heterogeneous Samples , 2008, IEEE Transactions on Signal Processing.

[5]  D. Pollock,et al.  Circulant matrices and time-series analysis , 2000 .

[6]  J. H. Steiger.  Tests for comparing elements of a correlation matrix. , 1980 .

[7]  M. Rao,et al.  Theory of Orlicz spaces , 1991 .

[8]  José M. F. Moura,et al.  Block matrices with L-block-banded inverse: inversion algorithms , 2005, IEEE Transactions on Signal Processing.

[9]  Alfred O. Hero,et al.  Distributed Covariance Estimation in Gaussian Graphical Models , 2010, IEEE Transactions on Signal Processing.

[10]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[11]  Amr Ahmed,et al.  Recovering time-varying networks of dependencies in social and biological studies , 2009, Proceedings of the National Academy of Sciences.

[12]  P. Massart,et al.  Adaptive estimation of a quadratic functional by model selection , 2000 .

[13]  S. Kay.  Fundamentals of statistical signal processing: estimation theory , 1993 .

[14]  Ami Wiesel,et al.  Time Varying Autoregressive Moving Average Models for Covariance Estimation , 2013, IEEE Transactions on Signal Processing.

[15]  P. Bickel,et al.  Regularized estimation of large covariance matrices , 2008, 0803.1909.

[16]  Raj Rao Nadakuditi,et al.  OptShrink: An Algorithm for Improved Low-Rank Signal Matrix Denoising by Optimal, Data-Driven Singular Value Shrinkage , 2013, IEEE Transactions on Information Theory.

[17]  Olivier Ledoit,et al.  Improved estimation of the covariance matrix of stock returns with an application to portfolio selection , 2003 .

[18]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[19]  Harrison H. Zhou,et al.  Optimal rates of convergence for estimating Toeplitz covariance matrices , 2013 .

[20]  R. Adamczak,et al.  Sharp bounds on the rate of convergence of the empirical covariance matrix , 2010, 1012.0294.

[21]  R. Maronna.  Robust $M$-Estimators of Multivariate Location and Scatter , 1976 .

[22]  Colin L. Mallows,et al.  Embedding nonnegative definite Toeplitz matrices in nonnegative definite circulant matrices, with application to covariance estimation , 1989, IEEE Trans. Inf. Theory.

[23]  D. Donoho,et al.  The Optimal Hard Threshold for Singular Values is 4 / √ 3 , 2013 .

[24]  Daniel R. Fuhrmann,et al.  Application of Toeplitz covariance estimation to adaptive beamforming and detection , 1991, IEEE Trans. Signal Process..

[25]  Ping-Cheng Yeh,et al.  An Interpretation of the Moore-Penrose Generalized Inverse of a Singular Fisher Information Matrix , 2011, IEEE Transactions on Signal Processing.

[26]  Jian Li,et al.  SPICE: A Sparse Covariance-Based Estimation Method for Array Processing , 2011, IEEE Transactions on Signal Processing.

[27]  Yuri I. Abramovich,et al.  Time-Varying Autoregressive (TVAR) Models for Multiple Radar Observations , 2007, IEEE Transactions on Signal Processing.

[28]  Jean-Yves Tourneret,et al.  A Bayesian Approach to Adaptive Detection in Nonhomogeneous Environments , 2008, IEEE Transactions on Signal Processing.

[29]  P. Priouret,et al.  On recursive estimation for time varying autoregressive processes , 2005, math/0603047.

[30]  M. Rothschild,et al.  Asset Pricing with a Factor Arch Covariance Structure: Empirical Estimates for Treasury Bills , 1988 .

[31]  S. Geer,et al.  The Bernstein–Orlicz norm and deviation inequalities , 2011, 1111.2450.

[32]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[33]  Kevin Sheppard,et al.  Evaluating Volatility and Correlation Forecasts , 2009 .

[34]  Patrick Danaher,et al.  The joint graphical lasso for inverse covariance estimation across multiple classes , 2011, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[35]  Peter Hördahl,et al.  Testing the Conditional CAPM Using Multivariate GARCH-M , 1998 .

[36]  Steven Kay,et al.  Fundamentals Of Statistical Signal Processing , 2001 .

[37]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[38]  K. Strimmer,et al.  Statistical Applications in Genetics and Molecular Biology A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics , 2011 .

[39]  V. Chandrasekaran,et al.  Group symmetry and covariance regularization , 2012 .

[40]  E. Levina,et al.  Joint estimation of multiple graphical models. , 2011, Biometrika.

[41]  Todd M. Allen,et al.  Coordinate linkage of HIV evolution reveals regions of immunological vulnerability , 2011, Proceedings of the National Academy of Sciences.

[42]  José M. F. Moura,et al.  Matrices with banded inverses: Inversion algorithms and factorization of Gauss-Markov processes , 2000, IEEE Trans. Inf. Theory.

[43]  G. W. Stewart,et al.  On the Early History of the Singular Value Decomposition , 1993, SIAM Rev..

[44]  Gongguo Tang,et al.  Lower Bounds on the Mean-Squared Error of Low-Rank Matrix Reconstruction , 2011, IEEE Transactions on Signal Processing.

[45]  William J. J. Roberts,et al.  Hidden Markov modeling of speech using Toeplitz covariance matrices , 2000, Speech Commun..

[46]  J. Wooldridge,et al.  A Capital Asset Pricing Model with Time-Varying Covariances , 1988, Journal of Political Economy.

[47]  David E. Tyler.  A Distribution-Free $M$-Estimator of Multivariate Scatter , 1987 .

[48]  J. Bouchaud,et al.  RANDOM MATRIX THEORY AND FINANCIAL CORRELATIONS , 2000 .

[49]  A. Hero,et al.  Robust shrinkage estimation of high-dimensional covariance matrices , 2010 .

[50]  Olivier Ledoit,et al.  Some hypothesis tests for the covariance matrix when the dimension is large compared to the sample size , 2002 .

[51]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[52]  Ami Wiesel,et al.  Proper Quaternion Gaussian Graphical Models , 2014, IEEE Transactions on Signal Processing.

[53]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[54]  David L. Donoho,et al.  The Optimal Hard Threshold for Singular Values is 4/sqrt(3) , 2013, 1305.5870.