OGM: Online gaussian graphical models on the fly

Gaussian Graphical Model is widely used to understand the dependencies between variables from high-dimensional data and can enable a wide range of applications such as principal component analysis, discriminant analysis, and canonical analysis. With respect to the streaming nature of big data, we study a novel Online Gaussian Graphical Model (OGM) that can estimate the inverse covariance matrix over the high-dimensional streaming data, in this paper. Specifically, given a small number of samples to initialize the learning process, OGM first estimates a low-rank estimation of inverse covariance matrix; then, when each individual new sample arrives, it updates the estimation of inverse covariance matrix using a low-complexity updating rule, without using the past data and matrix inverse. The significant edges of Gaussian graphical models can be discovered through thresholding the inverse covariance matrices. Theoretical analysis shows the convergence rate of OGM to the true parameters is guaranteed under Bernstein-style with mild conditions. We evaluate OGM using extensive experiments. The evaluation results backup our theory.

[1]  H. Zou,et al.  Positive Definite $\ell_1$ Penalized Estimation of Large Covariance Matrices , 2012, 1208.5702.

[2]  Yanjie Fu,et al.  MP2SDA: Multi-Party Parallelized Sparse Discriminant Learning , 2020, ACM Trans. Knowl. Discov. Data.

[3]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[4]  Jana Janková,et al.  Honest confidence regions and optimality in high-dimensional precision matrix estimation , 2015, TEST.

[5]  Julia M. Huntenburg,et al.  Loading and plotting of cortical surface representations in Nilearn , 2017 .

[6]  Joel A. Tropp,et al.  An Introduction to Matrix Concentration Inequalities , 2015, Found. Trends Mach. Learn..

[7]  Jianqing Fan,et al.  NETWORK EXPLORATION VIA THE ADAPTIVE LASSO AND SCAD PENALTIES. , 2009, The annals of applied statistics.

[8]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[9]  Wei Cheng,et al.  Sparse regression models for unraveling group and individual associations in eQTL mapping , 2016, BMC Bioinformatics.

[10]  MP2SDA , 2020, ACM Transactions on Knowledge Discovery from Data.

[11]  Kean Ming Tan,et al.  Sparse generalized eigenvalue problem: optimal statistical rates via truncated Rayleigh flow , 2016, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[12]  David Danks,et al.  Tracking Time-varying Graphical Structure , 2013, NIPS.

[13]  T. Cai,et al.  A Constrained ℓ1 Minimization Approach to Sparse Precision Matrix Estimation , 2011, 1102.2233.

[14]  Kshitij Khare,et al.  Posterior graph selection and estimation consistency for high-dimensional Bayesian DAG models , 2016, The Annals of Statistics.

[15]  Haoyi Xiong,et al.  Early detection of diseases using electronic health records data and covariance-regularized linear discriminant analysis , 2017, 2017 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[16]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[17]  Steffen Lauritzen,et al.  Gaussian Graphical Models , 2018, Handbook of Graphical Models.

[18]  Yanjie Fu,et al.  CRLEDD: Regularized Causalities Learning for Early Detection of Diseases Using Electronic Health Record (EHR) Data , 2021, IEEE Transactions on Emerging Topics in Computational Intelligence.

[19]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[20]  S. Geer,et al.  Confidence intervals for high-dimensional inverse covariance estimation , 2014, 1403.6752.

[21]  G. Casella,et al.  The Bayesian Lasso , 2008 .

[22]  Harrison H. Zhou,et al.  Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estimation , 2016 .

[23]  Judea Pearl,et al.  Graphical Models for Recovering Probabilistic and Causal Queries from Missing Data , 2014, NIPS.

[24]  T. Cai,et al.  A Direct Estimation Approach to Sparse Linear Discriminant Analysis , 2011, 1107.3442.

[25]  T. Cai,et al.  Sparse PCA: Optimal rates and adaptive estimation , 2012, 1211.1309.

[26]  Weidong Liu.  Gaussian graphical model estimation with false discovery rate control , 2013, 1306.0976.

[27]  Adam J. Rothman,et al.  Sparse permutation invariant covariance estimation , 2008, 0801.4837.

[28]  Lie Wang,et al.  Sparse Covariance Matrix Estimation With Eigenvalue Constraints , 2014, Journal of computational and graphical statistics : a joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America.

[29]  J. Pearl.  The structural theory of causation , 2011 .

[30]  Caroline Uhler,et al.  Gaussian Graphical Models: An Algebraic and Geometric Perspective , 2017, 1707.04345.

[31]  Hao Wang,et al.  Bayesian Graphical Lasso Models and Efficient Posterior Computation , 2012 .

[32]  Qiang Liu,et al.  Learning Scale Free Networks by Reweighted L1 regularization , 2011, AISTATS.

[33]  K. Miller.  On the Inverse of the Sum of Matrices , 1981 .

[34]  R. Tibshirani,et al.  Covariance‐regularized regression and classification for high dimensional problems , 2009, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[35]  Haoyi Xiong,et al.  Improving covariance-regularized discriminant analysis for EHR-based predictive analytics of diseases , 2020, Applied Intelligence.

[36]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[37]  Haoyi Xiong,et al.  $\mathcal{DBSDA}$ : Lowering the Bound of Misclassification Rate for Sparse Linear Discriminant Analysis via Model Debiasing , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[38]  Fang Han,et al.  Transelliptical Graphical Models , 2012, NIPS.

[39]  David L. Donoho,et al.  The Optimal Hard Threshold for Singular Values is 4/sqrt(3) , 2013, 1305.5870.

[40]  Xindong Wu,et al.  10 Challenging Problems in Data Mining Research , 2006, Int. J. Inf. Technol. Decis. Mak..

[41]  Kshitij Khare,et al.  High dimensional posterior convergence rates for decomposable graphical models , 2015 .

[42]  Bin Yu,et al.  High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence , 2008, 0811.3628.

[43]  Harrison H. Zhou,et al.  MINIMAX ESTIMATION OF LARGE COVARIANCE MATRICES UNDER ℓ1-NORM , 2012 .

[44]  Shonali Krishnaswamy,et al.  Mining data streams: a review , 2005, SGMD.

[45]  Deyuan Meng,et al.  Contraction Mapping-Based Robust Convergence of Iterative Learning Control With Uncertain, Locally Lipschitz Nonlinearity , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[46]  Jianhua Z. Huang,et al.  Covariance matrix selection and estimation via penalised normal likelihood , 2006 .