Maximum likelihood estimation of mixtures of factor analyzers

Mixtures of factor analyzers have been receiving wide interest in statistics as a tool for performing clustering and dimension reduction simultaneously. In this model it is assumed that, within each component, the data are generated according to a factor model. Therefore, the number of parameters on which the covariance matrices depend is reduced. Several estimation methods have been proposed for this model, both in the classical and in the Bayesian framework. However, so far, a direct maximum likelihood procedure has not been developed. This direct estimation problem, which simultaneously allows one to derive the information matrix for the mixtures of factor analyzers, is solved. The effectiveness of the proposed procedure is shown on a simulation study and on a toy example.

[1]  T. Louis.  Finding the Observed Information Matrix When Using the EM Algorithm , 1982 .

[2]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[3]  Jianhua Zhao,et al.  Fast ML Estimation for the Mixture of Factor Analyzers via an ECM Algorithm , 2008, IEEE Transactions on Neural Networks.

[4]  A. Utsugi,et al.  Bayesian Analysis of Mixtures of Factor Analyzers , 2001, Neural Computation.

[5]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[6]  Xinsheng Liu,et al.  The EM algorithm for the extended finite mixture of the factor analyzers model , 2008, Comput. Stat. Data Anal..

[7]  J. Wolfe.  PATTERN CLUSTERING BY MULTIVARIATE MIXTURE ANALYSIS. , 1970, Multivariate behavioral research.

[8]  Zoubin Ghahramani,et al.  Variational Inference for Bayesian Mixtures of Factor Analysers , 1999, NIPS.

[9]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[10]  M. Stephens.  Dealing with label switching in mixture models , 2000 .

[11]  N. E. Day.  Estimating the components of a mixture of normal distributions , 1969 .

[12]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[13]  Geoffrey J. McLachlan,et al.  Modelling high-dimensional data by mixtures of factor analyzers , 2003, Comput. Stat. Data Anal..

[14]  Jan R. Magnus,et al.  Maximum Likelihood Estimation of the Multivariate Normal Mixture Model , 2009 .

[15]  Geoffrey J. McLachlan,et al.  Mixtures of Factor Analyzers , 2000, International Conference on Machine Learning.

[16]  Xiao-Li Meng,et al.  Using EM to Obtain Asymptotic Variance-Covariance Matrices: The SEM Algorithm , 1991 .

[17]  Geoffrey J. McLachlan,et al.  Mixture models : inference and applications to clustering , 1989 .

[18]  D. M. Titterington,et al.  Mixtures of Factor Analysers. Bayesian Estimation and Inference by Stochastic Simulation , 2004, Machine Learning.

[19]  A. Raftery,et al.  Variable Selection for Model-Based Clustering , 2006 .

[20]  Xiao-Li Meng,et al.  The EM Algorithm—an Old Folk‐song Sung to a Fast New Tune , 1997 .

[21]  A. Goldman.  An Introduction to Regression Graphics , 1995 .

[22]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.