Learning a common substructure of multiple graphical Gaussian models

Properties of data are frequently seen to vary depending on the sampled situations, which usually change along a time evolution or owing to environmental effects. One way to analyze such data is to find invariances, or representative features kept constant over changes. The aim of this paper is to identify one such feature, namely interactions or dependencies among variables that are common across multiple datasets collected under different conditions. To that end, we propose a common substructure learning (CSSL) framework based on a graphical Gaussian model. We further present a simple learning algorithm based on the Dual Augmented Lagrangian and the Alternating Direction Method of Multipliers. We confirm the performance of CSSL over other existing techniques in finding unchanging dependency structures in multiple datasets through numerical simulations on synthetic data and through a real world application to anomaly detection in automobile sensors.

[1]  Christophe Ambroise,et al.  Inferring multiple graphical structures , 2009, Stat. Comput..

[2]  Martin J. Wainwright,et al.  High-Dimensional Graphical Model Selection Using ℓ1-Regularized Logistic Regression , 2006, NIPS.

[3]  Suvrit Sra,et al.  Fast Projections onto ℓ1, q -Norm Balls for Grouped Feature Selection , 2011, ECML/PKDD.

[4]  Jianqing Fan,et al.  NETWORK EXPLORATION VIA THE ADAPTIVE LASSO AND SCAD PENALTIES. , 2009, The annals of applied statistics.

[5]  H. Zou The Adaptive Lasso and Its Oracle Properties , 2006 .

[6]  Xiaoming Yuan,et al.  Alternating Direction Methods for Sparse Covariance Selection * , 2009 .

[7]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[8]  Stephen Gould,et al.  Projected Subgradient Methods for Learning Sparse Gaussians , 2008, UAI.

[9]  Dimitris Samaras,et al.  Multi-Task Learning of Gaussian Graphical Models , 2010, ICML.

[10]  Francis R. Bach,et al.  Consistency of the group Lasso and multiple kernel learning , 2007, J. Mach. Learn. Res..

[11]  K. Müller,et al.  Finding stationary subspaces in multivariate time series. , 2009, Physical review letters.

[12]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[13]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[14]  Naoki Abe,et al.  Proximity-Based Anomaly Detection Using Sparse Structure Learning , 2009, SDM.

[15]  Larry A. Wasserman,et al.  Time varying undirected graphs , 2008, Machine Learning.

[16]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[17]  Jean-Philippe Vert,et al.  Group Lasso with Overlaps: the Latent Group Lasso approach , 2011, ArXiv.

[18]  E. Levina,et al.  Joint estimation of multiple graphical models. , 2011, Biometrika.

[19]  M. Hestenes Multiplier and gradient methods , 1969 .

[20]  Yue Joseph Wang,et al.  Learning Structural Changes of Gaussian Graphical Models in Controlled Experiments , 2010, UAI.

[21]  Gene H. Golub,et al.  Matrix computations , 1983 .

[22]  Yoshinobu Kawahara,et al.  Separation of stationary and non-stationary sources with a generalized eigenvalue problem , 2012, Neural Networks.

[23]  Jean Honorio Lipschitz Parametrization of Probabilistic Graphical Models , 2011, UAI.

[24]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[25]  Pradeep Ravikumar,et al.  Sparse inverse covariance matrix estimation using quadratic approximation , 2011, MLSLP.

[26]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[27]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[28]  M. Yuan,et al.  Model selection and estimation in the Gaussian graphical model , 2007 .

[29]  Wenjie Fu,et al.  Recovering temporally rewiring networks: a model-based approach , 2007, ICML '07.

[30]  Robert Clarke,et al.  Differential dependency network analysis to identify condition-specific topological changes in biological networks , 2009, Bioinform..

[31]  Jean-Baptiste Poline,et al.  Brain covariance selection: better individual functional connectivity models using population prior , 2010, NIPS.

[32]  Kim-Chuan Toh,et al.  Solving Log-Determinant Optimization Problems by a Newton-CG Primal Proximal Point Algorithm , 2010, SIAM J. Optim..

[33]  Stephen J. Wright,et al.  Simultaneous Variable Selection , 2005, Technometrics.

[34]  B. Schölkopf,et al.  High-Dimensional Graphical Model Selection Using ℓ1-Regularized Logistic Regression , 2007 .

[35]  Masashi Sugiyama,et al.  Super-Linear Convergence of Dual Augmented Lagrangian Algorithm for Sparsity Regularized Estimation , 2009, J. Mach. Learn. Res..

[36]  Lu Li,et al.  An inexact interior point method for L1-regularized sparse covariance selection , 2010, Math. Program. Comput..

[37]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[38]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[39]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[40]  Katya Scheinberg,et al.  Learning Sparse Gaussian Markov Networks Using a Greedy Coordinate Ascent Approach , 2010, ECML/PKDD.

[41]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[42]  A. Willsky,et al.  Latent variable graphical model selection via convex optimization , 2010 .

[43]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[44]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[45]  Han Liu,et al.  Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery , 2009, ICML '09.

[46]  Melvin J. Hinich,et al.  Time Series Analysis by State Space Methods , 2001 .

[47]  Xiaorui Li Sparse Inverse Covariance Selection with Non-Convex Regularizations , 2013 .

[48]  Takashi Washio,et al.  Common Substructure Learning of Multiple Graphical Gaussian Models , 2011, ECML/PKDD.

[49]  R. Baillie,et al.  Common Stochastic Trends in a System of Exchange Rates , 1989 .

[50]  Amr Ahmed,et al.  Recovering time-varying networks of dependencies in social and biological studies , 2009, Proceedings of the National Academy of Sciences.

[51]  Martin J. Wainwright,et al.  Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions , 2011, ICML.

[52]  Shiqian Ma,et al.  Sparse Inverse Covariance Selection via Alternating Linearization Methods , 2010, NIPS.

[53]  Ali Jalali,et al.  A Dirty Model for Multi-task Learning , 2010, NIPS.

[54]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .