ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM

Two convergence aspects of the EM algorithm are studied: (i) does the EM algorithm find a local maximum or a stationary value of the (incompletedata) likelihood function? (ii) does the sequence of parameter estimates generated by EM converge? Several convergence results are obtained under conditions that are applicable to many practical situations. Two useful special cases are: (a) if the unobserved complete-data specification can be described by a curved exponential family with compact parameter space, all the limit points of any EM sequence are stationary points of the likelihood function; (b) if the likelihood function is unimodal and a certain differentiability condition is satisfied, then any EM sequence converges to the unique maximum likelihood estimate. A list of key properties of the algorithm is included.

[1]  Roland P. Falkner,et al.  History of statistics , 1891 .

[2]  K. Pearson.  Contributions to the Mathematical Theory of Evolution , 1894 .

[3]  J. I.  The Design of Experiments , 1936, Nature.

[4]  R. Fisher.  The Advanced Theory of Statistics , 1943, Nature.

[5]  Taylor Francis Online,et al.  The American statistician , 1947 .

[6]  Wilfred Perks,et al.  Some observations on inverse probability including a new indifference rule , 1947 .

[7]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[8]  E. S. Pearson,et al.  THE TIME INTERVALS BETWEEN INDUSTRIAL ACCIDENTS , 1952 .

[9]  J. Kiefer,et al.  Stochastic Estimation of the Maximum of a Regression Function , 1952 .

[10]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[11]  J. Tobin.  Estimation of Relationships for Limited Dependent Variables , 1958 .

[12]  A. Stuart.  Gamma-distributed products of independent random variables , 1962 .

[13]  M. Jöhnk.  Erzeugung von betaverteilten und gammaverteilten Zufallszahlen , 1964 .

[14]  H. Robbins.  The Empirical Bayes Approach to Statistical Decision Problems , 1964 .

[15]  G. Marsaglia.  Generating a Variable from the Tail of the Normal Distribution , 1964 .

[16]  M. Pike.  A method of analysis of a certain class of experiments in carcinogenesis. , 1966, Biometrics.

[17]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .

[18]  A. Ostrowski.  Solution of equations and systems of equations , 1967 .

[19]  V. Hasselblad.  Finite mixtures of distributions from the exponential family , 1969 .

[20]  Martin Pincus,et al.  Letter to the Editor - -A Closed Form Solution of Certain Programming Problems , 1968, Oper. Res..

[21]  T. Schoener.  The Anolis Lizards of Bimini: Resource Partitioning in a Complex Fauna , 1968 .

[22]  F. Downton.  Stochastic Approximation , 1969, Nature.

[23]  J. Wolfe.  PATTERN CLUSTERING BY MULTIVARIATE MIXTURE ANALYSIS. , 1970, Multivariate behavioral research.

[24]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[25]  E. M. L. Beale,et al.  Nonlinear Programming: A Unified Approach. , 1970 .

[26]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[27]  M. Rosenblatt.  Markov Processes, Structure and Asymptotic Behavior , 1971 .

[28]  S. Orey.  Lecture Notes on Limit Theorems for Markov Chain Transition Probabilities , 1971 .

[29]  R. R. Hocking,et al.  The analysis of incomplete data. , 1971 .

[30]  Elijah Polak,et al.  Computational methods in optimization , 1971 .

[31]  D. Vere-Jones.  Markov Chains , 1972, Nature.

[32]  M. Woodbury.  A missing information principle: theory and applications , 1972 .

[33]  P. Peskun,et al.  Optimum Monte-Carlo sampling using Markov chains , 1973 .

[34]  M. Stone,et al.  Marginalization Paradoxes in Bayesian and Structural Inference , 1973 .

[35]  John P. Moussouris.  Gibbs and Markov random systems with constraints , 1974 .

[36]  F. Olver.  Asymptotics and Special Functions , 1974 .

[37]  S. Haberman.  Log-Linear Models for Frequency Tables Derived by Indirect Observation: Maximum Likelihood Equations , 1974 .

[38]  G. N. Mil’shtejn.  Approximate Integration of Stochastic Differential Equations , 1975 .

[39]  B. Turnbull.  The Empirical Distribution Function with Arbitrarily Grouped, Censored, and Truncated Data , 1976 .

[40]  G. Marsaglia.  The squeeze method for generating gamma variates , 1977 .

[41]  S. Haberman.  Product Models for Frequency Tables Involving Indirect Observation , 1977 .

[42]  J. G. Ramage,et al.  Computer methods for sampling from student's t distribution , 1977 .

[43]  R. Davies.  Hypothesis testing when a nuisance parameter is present only under the alternative , 1977 .

[44]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[45]  S. Yakowitz,et al.  Weighted Monte Carlo Integration , 1978 .

[46]  A. F. Smith,et al.  A Quasi‐Bayes Sequential Procedure for Mixtures , 1978 .

[47]  N. Laird.  Nonparametric Maximum Likelihood Estimation of a Mixing Distribution , 1978 .

[48]  P. Diaconis,et al.  Conjugate Priors for Exponential Families , 1979 .

[49]  M. Hassell.  Capture-recapture methods , 1979, Nature.

[50]  B. Efron.  Computers and the Theory of Statistics: Thinking the Unthinkable , 1979 .

[51]  R. Jarrett.  A note on the intervals between coal-mining disasters , 1979 .

[52]  B. Schmeiser,et al.  Acceptance/Rejection Methods for Beta Variate Generation , 1980 .

[53]  R. Randles,et al.  Introduction to the Theory of Nonparametric Statistics , 1991 .

[54]  C. Hwang.  Laplace's Method Revisited: Weak Convergence of Probability Measures , 1980 .

[55]  Reuven Y. Rubinstein,et al.  Simulation and the Monte Carlo method , 1981, Wiley series in probability and mathematical statistics.

[56]  Charalambos D. Aliprantis,et al.  Principles of Real Analysis , 1981 .

[57]  I. Olkin,et al.  A Comparison of n Estimators for the Binomial Distribution , 1981 .

[58]  G. Wahba.  Spline Interpolation and Smoothing on the Sphere , 1981 .

[59]  G. Kallianpur.  Stochastic differential equations and diffusion processes , 1981 .

[60]  T. Louis.  Finding the Observed Information Matrix When Using the EM Algorithm , 1982 .

[61]  Tom Fearn,et al.  Contribution to discussion of paper by PJ Brown , 1982 .

[62]  Dorothy T. Thayer,et al.  EM algorithms for ML factor analysis , 1982 .

[63]  K. L. Saxena,et al.  Estimation of the Non-Centrality Parameter of a Chi Squared Distribution , 1982 .

[64]  T. Speed,et al.  Structural Analysis of Multivariate Data: A Review , 1982 .

[65]  Y. Vardi.  Nonparametric Estimation in Renewal Processes , 1982 .

[66]  J. Naylor,et al.  Applications of a Method for the Efficient Computation of Posterior Distributions , 1982 .

[67]  R. Dykstra,et al.  An Algorithm for Isotonic Regression for Two or More Independent Variables , 1982 .

[68]  R. A. Boyles.  On the Convergence of the EM Algorithm , 1983 .

[69]  H. Robbins.  Some Thoughts on Empirical Bayes Estimation , 1983 .

[70]  D. Rubin,et al.  On Jointly Estimating Parameters and Missing Data by Maximizing the Complete-Data Likelihood , 1983 .

[71]  B. Efron,et al.  The Jackknife: The Bootstrap and Other Resampling Plans. , 1983 .

[72]  H. Daniels.  Saddlepoint approximations for estimating equations , 1983 .

[73]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[74]  Lee W. Schruben,et al.  Optimal Tests for Initialization Bias in Simulation Output , 1983, Oper. Res..

[75]  Brian Everitt,et al.  An Introduction to Latent Variable Models , 1984 .

[76]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[77]  T. J. Mitchell,et al.  Nonparametric estimation of the distribution of time to onset for specific diseases in survival/sacrifice experiments. , 1984, Biometrics.

[78]  Yi-Ching Yao.  Estimation of a Noisy Discrete-Time Step Function: Bayes and Empirical Bayes Approaches , 1984 .

[79]  E. Nummelin.  General irreducible Markov chains and non-negative operators: Preface , 1984 .

[80]  A. Philippe.  Importance Sampling and Riemann Sums , 2022 .

[81]  Product Models , .

[82]  October I.  Physical Review Letters , 2022 .