Estimating Multivariate Latent-Structure Models

A constructive proof of identification of multilinear decompositions of multiway arrays is presented. It can be applied to show identification in a variety of multivariate latent structures. Examples are finite-mixture models and hidden Markov models. The key step to show identification is the joint diagonalization of a set of matrices in the same non-orthogonal basis. An estimator of the latent-structure model may then be based on a sample version of this simultaneous-diagonalization problem. Simple algorithms are available for computation. Asymptotic theory is derived for this joint approximate-diagonalization estimator.

[1]  Peter J. Diggle,et al.  The Selection of Terms in an Orthogonal Series Density Estimator , 1986 .

[2]  Lianfen Qian,et al.  Nonparametric Curve Estimation: Methods, Theory, and Applications , 1999, Technometrics.

[3]  B. Green,et al.  A general solution for the latent class model of latent structure analysis. , 1951, Psychometrika.

[4]  Harm Derksen,et al.  Kruskal's uniqueness inequality is sharp , 2013 .

[5]  T Petrie,et al.  Probabilistic functions of finite-state markov chains. , 1967, Proceedings of the National Academy of Sciences of the United States of America.

[6]  L. Chiantini,et al.  On generic identifiability of symmetric tensors of subgeneric rank , 2015, 1504.00547.

[7]  Lieven De Lathauwer,et al.  A Link between the Canonical Decomposition in Multilinear Algebra and Simultaneous Matrix Diagonalization , 2006, SIAM J. Matrix Anal. Appl..

[8]  P. Comon,et al.  Algebraic identification of under-determined mixtures , 2010 .

[9]  Elisabeth Gassiat,et al.  Finite state space non parametric Hidden Markov Models are in general identifiable , 2013, 1306.4657.

[10]  P. Deb.  Finite Mixture Models , 2008 .

[11]  Karim Abed-Meraim,et al.  A new Jacobi-like method for joint diagonalization of arbitrary non-defective matrices , 2009, Appl. Math. Comput..

[12]  Stéphane Robin,et al.  Inference in finite state space non parametric Hidden Markov Models and applications , 2016, Stat. Comput..

[13]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[14]  C. Matias,et al.  Parameter identifiability in a class of random graph mixture models , 2010, 1006.0826.

[15]  C. J. Stone,et al.  Optimal Global Rates of Convergence for Nonparametric Regression , 1982 .

[16]  Giorgio Ottaviani,et al.  An Algorithm For Generic and Low-Rank Specific Identifiability of Complex Tensors , 2014, SIAM J. Matrix Anal. Appl..

[17]  N. Sidiropoulos,et al.  On the uniqueness of multilinear decomposition of N‐way arrays , 2000 .

[18]  Haikady N. Nagaraja,et al.  Inference in Hidden Markov Models , 2006, Technometrics.

[19]  Karim Abed-Meraim,et al.  Joint diagonalization of non defective matrices using generalized Jacobi rotations , 2010, 10th International Conference on Information Science, Signal Processing and their Applications (ISSPA 2010).

[20]  Pierre Comon,et al.  Handbook of Blind Source Separation: Independent Component Analysis and Applications , 2010 .

[21]  Xiqi Gao,et al.  Simultaneous Diagonalization With Similarity Transformation for Non-Defective Matrices , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[22]  Lieven De Lathauwer,et al.  Canonical Polyadic Decomposition of Third-Order Tensors: Reduction to Generalized Eigenvalue Decomposition , 2013, SIAM J. Matrix Anal. Appl..

[23]  A. Azzalini.  A class of distributions which includes the normal ones , 1985 .

[24]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part III: Alternating Least Squares Algorithms , 2008, SIAM J. Matrix Anal. Appl..

[25]  D. Hunter,et al.  Inference for mixtures of symmetric distributions , 2007, 0708.0499.

[26]  L. Bordes,et al.  SEMIPARAMETRIC ESTIMATION OF A TWO-COMPONENT MIXTURE MODEL , 2006, math/0607812.

[27]  Xiao-Hua Zhou,et al.  NONPARAMETRIC ESTIMATION OF COMPONENT DISTRIBUTIONS IN A MULTIVARIATE MIXTURE , 2003 .

[28]  M. Powell,et al.  Approximation theory and methods , 1984 .

[29]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[30]  P. Hall,et al.  Nonparametric inference in multivariate mixtures , 2005 .

[31]  Lieven De Lathauwer,et al.  Generic Uniqueness Conditions for the Canonical Polyadic Decomposition and INDSCAL , 2014, SIAM J. Matrix Anal. Appl..

[32]  Nikos D. Sidiropoulos,et al.  Kruskal's permutation lemma and the identification of CANDECOMP/PARAFAC and bilinear models with constant modulus constraints , 2004, IEEE Transactions on Signal Processing.

[33]  Joos Vandewalle,et al.  Computation of the Canonical Decomposition by Means of a Simultaneous Generalized Schur Decomposition , 2005, SIAM J. Matrix Anal. Appl..

[34]  Thomas P. Hettmansperger,et al.  Almost nonparametric inference for repeated measures in mixture models , 2000 .

[35]  Jean-Marc Robin,et al.  Non‐parametric estimation of finite mixtures from repeated measurements , 2016 .

[36]  C. Matias,et al.  Identifiability of parameters in latent structure models with many observed variables , 2008, 0809.5032.

[37]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[38]  Lieven De Lathauwer,et al.  On the Uniqueness of the Canonical Polyadic Decomposition of Third-Order Tensors - Part I: Basic Results and Uniqueness of One Factor Matrix , 2013, SIAM J. Matrix Anal. Appl..

[39]  J. Kruskal.  Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[40]  W. Newey,et al.  Convergence rates and asymptotic normality for series estimators , 1997 .

[41]  W. Newey,et al.  Large sample estimation and hypothesis testing , 1986 .

[42]  J. Kruskal.  More factors than subjects, tests and treatments: An indeterminacy theorem for canonical decomposition and individual differences scaling , 1976 .

[43]  Laurent Albera,et al.  Joint Eigenvalue Decomposition Using Polar Matrix Factorization , 2010, LVA/ICA.

[44]  Leo A. Goodman,et al.  On the estimation of parameters in latent structure analysis , 1979 .

[45]  On the asymptotic theory for least squares series : pointwise and uniform results , 2013 .

[46]  H. Kasahara,et al.  Nonparametric Identification of Finite Mixture Models of Dynamic Discrete Choices , 2009 .

[47]  E. Liebscher.  Hermite series estimators for probability densities , 1990 .

[48]  Bin Yu,et al.  Spectral clustering and the high-dimensional stochastic blockmodel , 2010, 1007.1684.

[49]  P. Hall.  Cross—validation and the smoothing of orthogonal series density estimators , 1987 .

[50]  K. Roeder.  A Graphical Technique for Determining the Number of Components in a Mixture of Normals , 1994 .

[51]  X. Luciani,et al.  Canonical Polyadic Decomposition based on joint eigenvalue decomposition , 2014 .

[52]  B. Salani'e,et al.  Inference on Mixtures Under Tail Restrictions , 2013 .

[53]  H. Kasahara,et al.  Non‐parametric identification and estimation of the number of components in multivariate mixtures , 2014 .

[54]  T. Snijders,et al.  Estimation and Prediction for Stochastic Blockmodels for Graphs with Latent Block Structure , 1997 .

[55]  Lieven De Lathauwer,et al.  On the Uniqueness of the Canonical Polyadic Decomposition of Third-Order Tensors - Part II: Uniqueness of the Overall Decomposition , 2013, SIAM J. Matrix Anal. Appl..

[56]  David R. Hunter,et al.  An EM-Like Algorithm for Semi- and Nonparametric Estimation in Multivariate Mixtures , 2009 .

[57]  S. Schwartz.  Estimation of Probability Density by an Orthogonal Series , 1967 .

[58]  Pierre Comon,et al.  Canonical Polyadic Decomposition with a Columnwise Orthonormal Factor Matrix , 2012, SIAM J. Matrix Anal. Appl..

[59]  D. Hunter,et al.  Maximum smoothed likelihood for multivariate mixtures , 2011 .