Fast ML Estimation for the Mixture of Factor Analyzers via an ECM Algorithm

In this brief, we propose a fast expectation conditional maximization (ECM) algorithm for maximum-likelihood (ML) estimation of mixtures of factor analyzers (MFA). Unlike the existing expectation-maximization (EM) algorithms such as the EM in Ghahramani and Hinton, 1996, and the alternating ECM (AECM) in McLachlan and Peel, 2003, where the missing data contains component-indicator vectors as well as latent factors, the missing data in our ECM consists of component-indicator vectors only. The novelty of our algorithm is that closed-form expressions in all conditional maximization (CM) steps are obtained explicitly, instead of resorting to numerical optimization methods. As revealed by experiments, the convergence of our ECM is substantially faster than EM and AECM regardless of whether assessed by central processing unit (CPU) time or number of iterations.

[1]  Kaare Brandt Petersen,et al.  On the Slow Convergence of EM and VBEM in Low-Noise Linear Models , 2005, Neural Computation.

[2]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[3]  D. Rubin,et al.  The ECME algorithm: A simple extension of EM and ECM with faster monotone convergence , 1994 .

[4]  Geoffrey J. McLachlan,et al.  Modelling high-dimensional data by mixtures of factor analyzers , 2003, Comput. Stat. Data Anal..

[5]  Xiao-Li Meng,et al.  The EM Algorithm—an Old Folk‐song Sung to a Fast New Tune , 1997 .

[6]  Xiao-Li Meng,et al.  Maximum likelihood estimation via the ECM algorithm: A general framework , 1993 .

[7]  Jianhua Zhao,et al.  ML estimation for factor analysis: EM or non-EM? , 2008, Stat. Comput..

[8]  Geoffrey J. McLachlan,et al.  Extension of the mixture of factor analyzers model to incorporate the multivariate t-distribution , 2007, Comput. Stat. Data Anal..

[9]  K. Jöreskog.  Some contributions to maximum likelihood factor analysis , 1967 .

[10]  Kenneth Lange,et al.  Numerical analysis for statisticians , 1999 .

[11]  Dorothy T. Thayer,et al.  EM algorithms for ML factor analysis , 1982 .

[12]  Donald B. Rubin,et al.  Max-imum Likelihood from Incomplete Data , 1972 .

[13]  Geoffrey E. Hinton,et al.  SMEM Algorithm for Mixture Models , 1998, Neural Computation.

[14]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .