An inexact interior point method for L1-regularized sparse covariance selection

Sparse covariance selection problems can be formulated as log-determinant (log-det) semidefinite programming (SDP) problems with large numbers of linear constraints. Standard primal–dual interior-point methods that are based on solving the Schur complement equation would encounter severe computational bottlenecks if they are applied to solve these SDPs. In this paper, we consider a customized inexact primal–dual path-following interior-point algorithm for solving large scale log-det SDP problems arising from sparse covariance selection problems. Our inexact algorithm solves the large and ill-conditioned linear system of equations in each iteration by a preconditioned iterative solver. By exploiting the structures in sparse covariance selection problems, we are able to design highly effective preconditioners to efficiently solve the large and ill-conditioned linear systems. Numerical experiments on both synthetic and real covariance selection problems show that our algorithm is highly efficient and outperforms other existing algorithms.

[1]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[2]  J. N. R. Jeffers,et al.  Graphical Models in Applied Multivariate Statistics. , 1990 .

[3]  R. Freund,et al.  A new Krylov-subspace method for symmetric indefinite linear systems , 1994 .

[4]  D. Edwards.  Introduction to graphical modelling , 1995 .

[5]  Kim-Chuan Toh,et al.  SDPT3 -- A Matlab Software Package for Semidefinite Programming , 1996 .

[6]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[7]  A. Wathen,et al.  The convergence of iterative solution methods for symmetric and indefinite linear systems , 1997 .

[8]  Michael J. Todd,et al.  Primal-Dual Interior-Point Methods for Self-Scaled Cones , 1998, SIAM J. Optim..

[9]  Yin Zhang,et al.  On Extending Some Primal-Dual Interior-Point Algorithms From Linear Programming to Semidefinite Programming , 1998, SIAM J. Optim..

[10]  Stephen P. Boyd,et al.  Determinant Maximization with Linear Matrix Inequality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[11]  Kim-Chuan Toh,et al.  On the Nesterov-Todd Direction in Semidefinite Programming , 1998, SIAM J. Optim..

[12]  J. Mesirov,et al.  Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. , 1999, Science.

[13]  Jeff A. Bilmes,et al.  Natural statistical models for automatic speech recognition , 1999 .

[14]  Ramesh A. Gopinath,et al.  Model selection in acoustic modeling , 1999, EUROSPEECH.

[15]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[16]  Kim-Chuan Toh,et al.  SDPT3 — a Matlab software package for semidefinite-quadratic-linear programming, version 3.0 , 2001 .

[17]  E. Dougherty,et al.  Gene-expression profiles in hereditary breast cancer. , 2001, The New England journal of medicine.

[18]  Yin Zhang,et al.  A computational study of a gradient-based log-barrier algorithm for a class of large-scale SDPs , 2003, Math. Program..

[19]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[20]  M. Pourahmadi,et al.  Nonparametric estimation of large covariance matrices of longitudinal data , 2003 .

[21]  R. Kohn,et al.  Efficient estimation of covariance selection models , 2003 .

[22]  John D. Storey,et al.  Statistical significance for genomewide studies , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Kim-Chuan Toh,et al.  Polynomiality of an inexact infeasible interior point algorithm for semidefinite programming , 2004, Math. Program..

[24]  M. West,et al.  Integrated modeling of clinical and gene expression information for personalized prediction of disease outcomes. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[25]  Kim-Chuan Toh,et al.  Solving Large Scale Semidefinite Programs via an Iterative Solver on the Augmented Systems , 2003, SIAM J. Optim..

[26]  M. West,et al.  Sparse graphical models for exploring gene expression data , 2004 .

[27]  P. Bühlmann,et al.  Sparse graphical Gaussian modeling of the isoprenoid gene network in Arabidopsis thaliana , 2004, Genome Biology.

[28]  K. Sachs,et al.  Causal Protein-Signaling Networks Derived from Multiparameter Single-Cell Data , 2005, Science.

[29]  Adrian E. Raftery,et al.  Bayesian model averaging: development of an improved multi-class, gene selection and classification tool for microarray data , 2005, Bioinform..

[30]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[31]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[32]  Franz Rendl,et al.  A Boundary Point Method to Solve Semidefinite Programs , 2006, Computing.

[33]  T. Tsuchiya,et al.  An extension of the standard polynomial-time primal-dual path-following algorithm to the weighted determinant maximization problem with semidefinite constraints , 2006 .

[34]  M. Yuan,et al.  Model selection and estimation in the Gaussian graphical model , 2007 .

[35]  Alexandre d'Aspremont,et al.  Identifying small mean-reverting portfolios , 2007, ArXiv.

[36]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Maximum Likelihood Estimation , 2007, ArXiv.

[37]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[38]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[39]  Kim-Chuan Toh,et al.  An inexact primal–dual path following algorithm for convex quadratic SDP , 2007, Math. Program..

[40]  Vwani P. Roychowdhury,et al.  Covariance selection for nonchordal graphs via chordal embedding , 2008, Optim. Methods Softw..

[41]  Alexandre d'Aspremont,et al.  First-Order Methods for Sparse Covariance Selection , 2006, SIAM J. Matrix Anal. Appl..

[42]  Franz Rendl,et al.  An Augmented Primal-Dual Method for Linear Conic Programs , 2008, SIAM J. Optim..

[43]  A. Dobra.  Dependency networks for genome-wide data , 2008 .

[44]  A. Dobra.  Variable selection and dependency networks for genomewide data. , 2009, Biostatistics.

[45]  Katya Scheinberg,et al.  IBM Research Report SINCO - A Greedy Coordinate Ascent Method for Sparse Inverse Covariance Selection Problem , 2009 .

[46]  T. Tsuchiya,et al.  Covariance regularization in inverse space , 2009 .

[47]  O. SIAMJ..  SMOOTH OPTIMIZATION APPROACH FOR SPARSE COVARIANCE SELECTION∗ , 2009 .

[48]  Xiaoming Yuan,et al.  Alternating Direction Methods for Sparse Covariance Selection * , 2009 .

[49]  Jianqing Fan,et al.  NETWORK EXPLORATION VIA THE ADAPTIVE LASSO AND SCAD PENALTIES. , 2009, The annals of applied statistics.

[50]  A. d'Aspremont,et al.  A Pathwise Algorithm for Covariance Selection , 2009, 0908.0143.

[51]  Zhaosong Lu,et al.  Adaptive First-Order Methods for General Sparse Inverse Covariance Selection , 2009, SIAM J. Matrix Anal. Appl..

[52]  Kim-Chuan Toh,et al.  A Newton-CG Augmented Lagrangian Method for Semidefinite Programming , 2010, SIAM J. Optim..

[53]  Kim-Chuan Toh,et al.  Solving Log-Determinant Optimization Problems by a Newton-CG Primal Proximal Point Algorithm , 2010, SIAM J. Optim..

[54]  Katya Scheinberg,et al.  Learning Sparse Gaussian Markov Networks Using a Greedy Coordinate Ascent Approach , 2010, ECML/PKDD.

[55]  Renato D. C. Monteiro,et al.  Primal-dual first-order methods with $${\mathcal {O}(1/\epsilon)}$$ iteration-complexity for cone programming , 2011, Math. Program..

[56]  Guanghui Lan,et al.  Primal-dual first-order methods with O (1/e) iteration-complexity for cone programming. , 2011 .

[57]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.