Parameter expansion to accelerate EM: The PX-EM algorithm

SUMMARY The EM algorithm and its extensions are popular tools for modal estimation but ar-e often criticised for their slow convergence. We propose a new method that can often make EM much faster. The intuitive idea is to use a 'covariance adjustment' to correct the analysis of the M step, capitalising on extra information captured in the imputed complete data. The way we accomplish this is by parameter expansion; we expand the complete-data model while preserving the observed-data model and use the expanded complete-data model to generate EM. This parameter-expanded Ei M, PX-EM, algorithm shares the simplicity and stability of ordinary EM, but has a faster rate of convergence since its M step performs a more efficient analysis. The PX-EM algorithm is illustrated for the multivariate t distribution, a random effects model, factor analysis, probit regression and a Poisson imaging model.

[1]  H. Hartley,et al.  Maximum-likelihood estimation for the mixed analysis of variance model. , 1967, Biometrika.

[2]  J. Ware,et al.  Random-effects models for longitudinal data. , 1982, Biometrics.

[3]  Dorothy T. Thayer,et al.  EM algorithms for ML factor analysis , 1982 .

[4]  L. Shepp,et al.  Maximum Likelihood Reconstruction for Emission Tomography , 1983, IEEE Transactions on Medical Imaging.

[5]  New York Dover,et al.  ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM , 1983 .

[6]  L. Shepp,et al.  A Statistical Model for Positron Emission Tomography , 1985 .

[7]  A. F. Smith,et al.  Statistical analysis of finite mixture distributions , 1986 .

[8]  N. Laird,et al.  Maximum likelihood computations with repeated measures: application of the EM algorithm , 1987 .

[9]  D. Rubin.  Multiple imputation for nonresponse in surveys , 1989 .

[10]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[11]  D. Bates,et al.  Newton-Raphson and EM Algorithms for Linear Mixed-Effects Models for Repeated-Measures Data , 1988 .

[12]  Jeremy MG Taylor,et al.  Robust Statistical Modeling Using the t Distribution , 1989 .

[13]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[14]  Trevor Hastie,et al.  Statistical Models in S , 1991 .

[15]  R. Jennrich,et al.  Conjugate Gradient Acceleration of the EM Algorithm , 1993 .

[16]  Xiao-Li Meng,et al.  Maximum likelihood estimation via the ECM algorithm: A general framework , 1993 .

[17]  Alfred O. Hero,et al.  Space-alternating generalized expectation-maximization algorithm , 1994, IEEE Trans. Signal Process..

[18]  David E. Tyler,et al.  A curious likelihood identity for the multivariate t-distribution , 1994 .

[19]  D. Rubin,et al.  The ECME algorithm: A simple extension of EM and ECM with faster monotone convergence , 1994 .

[20]  Jun S. Liu,et al.  Covariance structure of the Gibbs sampler with applications to the comparisons of estimators and augmentation schemes , 1994 .

[21]  K. Lange.  A gradient algorithm locally equivalent to the EM algorithm , 1995 .

[22]  J. Kent,et al.  Convergence Behavior of the em algorithm for the multivariate t -distribution , 1995 .

[23]  Chuanhai Liu.  ML Estimation of the MultivariatetDistribution and the EM Algorithm , 1997 .

[24]  Xiao-Li Meng,et al.  The EM Algorithm—an Old Folk‐song Sung to a Fast New Tune , 1997 .

[25]  D. Rubin,et al.  MAXIMUM LIKELIHOOD ESTIMATION OF FACTOR ANALYSIS USING THE ECME ALGORITHM WITH COMPLETE AND INCOMPLETE DATA , 1998 .

[26]  Xiao-Li Meng,et al.  Fast EM‐type implementations for mixed effects models , 1998 .