Accelerating the quadratic lower-bound algorithm via optimizing the shrinkage parameter

When the Newton-Raphson algorithm or the Fisher scoring algorithm does not work and the EM-type algorithms are not available, the quadratic lower-bound (QLB) algorithm may be a useful optimization tool. However, like all EM-type algorithms, the QLB algorithm may also suffer from slow convergence which can be viewed as the cost for having the ascent property. This paper proposes a novel 'shrinkage parameter' approach to accelerate the QLB algorithm while maintaining its simplicity and stability (i.e., monotonic increase in log-likelihood). The strategy is first to construct a class of quadratic surrogate functions Q"r(@q|@q^(^t^)) that induces a class of QLB algorithms indexed by a 'shrinkage parameter' r (r@?R) and then to optimize r over R under some criterion of convergence. For three commonly used criteria (i.e., the smallest eigenvalue, the trace and the determinant), we derive a uniformly optimal shrinkage parameter and find an optimal QLB algorithm. Some theoretical justifications are also presented. Next, we generalize the optimal QLB algorithm to problems with penalizing function and then investigate the associated properties of convergence. The optimal QLB algorithm is applied to fit a logistic regression model and a Cox proportional hazards model. Two real datasets are analyzed to illustrate the proposed methods.

[1]  Masahiro Kuroda,et al.  Accelerating the convergence of the EM algorithm using the vector epsilon , 2006, Comput. Stat. Data Anal..

[2]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[3]  D. Harrington,et al.  Counting Processes and Survival Analysis , 1991 .

[4]  B. Lindsay,et al.  Monotonicity of quadratic-approximation algorithms , 1988 .

[5]  David R. Cox,et al.  Regression models and life tables (with discussion , 1972 .

[6]  Xiao-Li Meng,et al.  Maximum likelihood estimation via the ECM algorithm: A general framework , 1993 .

[7]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[8]  Xiao-Li Meng,et al.  Fast EM‐type implementations for mixed effects models , 1998 .

[9]  Alfred O. Hero,et al.  Space-alternating generalized expectation-maximization algorithm , 1994, IEEE Trans. Signal Process..

[10]  D. Cox,et al.  Analysis of Survival Data. , 1985 .

[11]  Kenneth Lange,et al.  Numerical analysis for statisticians , 1999 .

[12]  Elisa T. Lee,et al.  A computer program for linear logistic regression analysis. , 1974, Computer programs in biomedicine.

[13]  D. Rubin,et al.  Parameter expansion to accelerate EM: The PX-EM algorithm , 1998 .

[14]  A FAST EM ALGORITHM FOR QUADRATIC OPTIMIZATION SUBJECT TO CONVEX CONSTRAINTS , 2007 .

[15]  Alain Berlinet,et al.  Parabolic acceleration of the EM algorithm , 2009, Stat. Comput..

[16]  Hua Zhou,et al.  A quasi-Newton acceleration for high-dimensional optimization algorithms , 2011, Stat. Comput..

[17]  D. V. Dyk,et al.  Fitting Mixed-Effects Models Using Efficient EM-Type Algorithms , 2000 .

[18]  P. Green.  On Use of the EM Algorithm for Penalized Likelihood Estimation , 1990 .

[19]  Xiao-Li Meng,et al.  Using EM to Obtain Asymptotic Variance-Covariance Matrices: The SEM Algorithm , 1991 .

[20]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[21]  Dankmar Böhning,et al.  The lower bound method in probit regression , 1999 .

[22]  R. Jennrich,et al.  Acceleration of the EM Algorithm by using Quasi‐Newton Methods , 1997 .

[23]  Richard A. Tapia,et al.  A trust region strategy for nonlinear equality constrained op-timization , 1984 .

[24]  Xiao-Li Meng,et al.  The EM Algorithm—an Old Folk‐song Sung to a Fast New Tune , 1997 .

[25]  R. Tibshirani.  The lasso method for variable selection in the Cox model. , 1997, Statistics in medicine.

[26]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[27]  J. Ortega.  Numerical Analysis: A Second Course , 1974 .

[28]  Masahiro Kuroda,et al.  Acceleration of the EM algorithm using the vector epsilon algorithm , 2008, Comput. Stat..

[29]  Xiao-Li Meng,et al.  On the rate of convergence of the ECM algorithm , 1994 .

[30]  D. Hunter,et al.  Optimization Transfer Using Surrogate Objective Functions , 2000 .

[31]  D. Hunter,et al.  A Tutorial on MM Algorithms , 2004 .

[32]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[33]  D. Böhning.  Multinomial logistic regression algorithm , 1992 .

[34]  R. Varadhan,et al.  Simple and Globally Convergent Methods for Accelerating the Convergence of Any EM Algorithm , 2008 .

[35]  R. Jennrich,et al.  Conjugate Gradient Acceleration of the EM Algorithm , 1993 .