TENSOR DECOMPOSITIONS AND SPARSE LOG-LINEAR MODELS.

Contingency table analysis routinely relies on log-linear models, with latent structure analysis providing a common alternative. Latent structure models lead to a reduced rank tensor factorization of the probability mass function for multivariate categorical data, while log-linear models achieve dimensionality reduction through sparsity. Little is known about the relationship between these notions of dimensionality reduction in the two paradigms. We derive several results relating the support of a log-linear model to nonnegative ranks of the associated probability tensor. Motivated by these findings, we propose a new collapsed Tucker class of tensor decompositions, which bridge existing PARAFAC and Tucker decompositions, providing a more flexible framework for parsimoniously characterizing multivariate categorical data. Taking a Bayesian approach to inference, we illustrate empirical advantages of the new decompositions.

[1]  Helene Massam,et al.  Bayes factors and the geometry of discrete hierarchical loglinear models , 2011, 1103.5381.

[2]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[3]  Stephen E. Fienberg,et al.  Discrete Multivariate Analysis: Theory and Practice , 1976 .

[4]  D. Dunson,et al.  Simplex Factor Models for Multivariate Unordered Categorical Data , 2012, Journal of the American Statistical Association.

[5]  A. Madansky.  Determinantal methods in latent class analysis , 1960 .

[6]  Bernd Sturmfels,et al.  Algebraic geometry of Bayesian networks , 2005, J. Symb. Comput..

[7]  T. W. Anderson.  On estimation of parameters in latent structure analysis , 1954 .

[8]  Dan Geiger,et al.  Asymptotic Model Selection for Naive Bayesian Networks , 2002, J. Mach. Learn. Res..

[9]  Jim Q. Smith,et al.  On the Geometry of Bayesian Graphical Models with Hidden Variables , 1998, UAI.

[10]  W. Gibson.  An extension of Anderson's solution for the latent structure equations , 1955 .

[11]  H. Massam,et al.  A conjugate prior for discrete hierarchical log-linear models , 2006, 0711.1609.

[12]  David Dunson,et al.  Bayesian Factorizations of Big Sparse Tensors , 2013, Journal of the American Statistical Association.

[13]  P. Dellaportas,et al.  Markov chain Monte Carlo model determination for hierarchical and graphical log-linear models , 1999 .

[14]  D. Dunson,et al.  Nonparametric Bayes Modeling of Multivariate Categorical Data , 2009, Journal of the American Statistical Association.

[15]  Tsuyoshi Kunihama,et al.  Bayesian Modeling of Temporal Dependence in Large Sparse Contingency Tables , 2012, Journal of the American Statistical Association.

[16]  T. Speed,et al.  Markov Fields and Log-Linear Interaction Models for Contingency Tables , 1980 .

[17]  A. Dobra,et al.  Copula Gaussian graphical models and their application to modeling functional disability data , 2011, 1108.1680.

[18]  M. West,et al.  Sparse graphical models for exploring gene expression data , 2004 .

[19]  Jim Q. Smith,et al.  Bayesian networks for discrete multivariate data: an algebraic approach to inference , 2003 .

[20]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[21]  Paul F. Lazarsfeld,et al.  Latent Structure Analysis. , 1969 .

[22]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[23]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[24]  Pierre Comon,et al.  Nonnegative approximations of nonnegative tensors , 2009, ArXiv.

[25]  Volker Roth,et al.  The Group-Lasso for generalized linear models: uniqueness of solutions and efficient algorithms , 2008, ICML '08.

[26]  H. Massam,et al.  The mode oriented stochastic search (MOSS) algorithm for log-linear models with conjugate priors , 2010 .

[27]  Alan Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[28]  Stephen E. Fienberg,et al.  Three centuries of categorical data analysis: Log-linear models and maximum likelihood estimation , 2007 .

[29]  Neil Henry.  Latent structure analysis , 1969 .

[30]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[31]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[32]  L. A. Goodman.  Exploratory latent structure analysis using both identifiable and unidentifiable models , 1974 .

[33]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[34]  Joel E. Cohen,et al.  Nonnegative ranks, decompositions, and factorizations of nonnegative matrices , 1993 .

[35]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[36]  D. Titterington,et al.  Variational approximations for categorical causal modeling with latent variables , 2003 .

[37]  A. Rinaldo,et al.  The Log-Linear Group Lasso Estimator and Its Asymptotic Properties , 2007, 0709.3526.

[38]  D. Geiger,et al.  Stratified exponential families: Graphical models and model selection , 2001 .

[39]  S. Haberman.  Log-Linear Models for Frequency Tables Derived by Indirect Observation: Maximum Likelihood Equations , 1974 .

[40]  A. Dawid,et al.  Hyper Markov Laws in the Statistical Analysis of Decomposable Graphical Models , 1993 .

[41]  S. E. Fienberg,et al.  Maximum Likelihood Estimation in Latent Class Models For Contingency Table Data , 2007, 0709.3535.

[42]  S. Stouffer,et al.  Measurement and Prediction , 1954 .