Properties of MM Algorithms on Convex Feasible Sets : Extended Version

We examine some properties of the Majorize-Minimize (MM) optimization technique, generalizing previous analyses. At each iteration of an MM algorithm, one constructs a tangent majorant function that majorizes the given cost function (possibly after adding a global constant) and is equal to it at the current iterate. The next iterate is taken from the set of minimizers of this tangent majorant function, resulting in a sequence of iterates that reduces the cost function monotonically. The article studies the behavior of these algorithms for problems with convex feasible sets but possibly non-convex cost functions. We analyze convergence properties in a standard way, showing first that the iteration sequence has stationary limit points under fairly mild conditions. We then obtain convergence results by adding discreteness assumptions on the stationary points of the minimization problem. The case where the stationary points form continua is also examined. Local convergence results are also developed for algorithms that use connected (e.g., convex) tangent majorants. Such algorithms have the property that the iterates cannot leave any basin-like region containing the initial vector. This makes MM useful in various non-convex minimization strategies that involve basin-probing steps. This property also implies that cost function minimizers will locally attract the iterates over larger neighborhoods than can typically be guaranteed with other methods. Our analysis generalizes previous work in several respects. Firstly, arbitrary convex feasible sets are permitted. The tangent majorant domains are also assumed convex, however they can be strict subsets of the feasible set. Secondly, the cost function and the tangent majorant functions are not required to be more than once continuously differentiable and the tangent majorants are often allowed to be non-convex as well. Thirdly, the technique of coordinate block alternation is considered for feasible sets of a more general Cartesian product form than in previous work.

[1]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[2]  A. Ostrowski Solution of equations in Euclidean and Banach spaces , 1973 .

[3]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[4]  L. Shepp,et al.  Maximum Likelihood Reconstruction for Emission Tomography , 1983, IEEE Transactions on Medical Imaging.

[5]  New York Dover,et al.  ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM , 1983 .

[6]  K. Lange,et al.  EM reconstruction algorithms for emission and transmission tomography. , 1984, Journal of computer assisted tomography.

[7]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[8]  P. Tseng,et al.  On the convergence of the coordinate descent method for convex differentiable minimization , 1992 .

[9]  A. R. De Pierro,et al.  On the relation between the ISRA and the EM algorithm for positron emission tomography , 1993, IEEE Trans. Medical Imaging.

[10]  Jeffrey A. Fessler,et al.  On complete-data spaces for PET reconstruction algorithms , 1993 .

[11]  H. Malcolm Hudson,et al.  Accelerated image reconstruction using ordered subsets of projection data , 1994, IEEE Trans. Medical Imaging.

[12]  Alfred O. Hero,et al.  Space-alternating generalized expectation-maximization algorithm , 1994, IEEE Trans. Signal Process..

[13]  K. Lange A gradient algorithm locally equivalent to the EM algorithm , 1995 .

[14]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[15]  Alfred O. Hero,et al.  Ieee Transactions on Image Processing: to Appear Penalized Maximum-likelihood Image Reconstruction Using Space-alternating Generalized Em Algorithms , 2022 .

[16]  Alvaro R. De Pierro,et al.  On the convergence of an EM-type algorithm for penalized likelihood estimation in emission tomography , 1995, IEEE Trans. Medical Imaging.

[17]  Alvaro R. De Pierro,et al.  A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography , 1995, IEEE Trans. Medical Imaging.

[18]  Xiao-Li Meng,et al.  The EM Algorithm—an Old Folk‐song Sung to a Fast New Tune , 1997 .

[19]  Dan Nettleton,et al.  Convergence properties of the EM algorithm in constrained parameter spaces , 1999 .

[20]  Jeffrey A. Fessler,et al.  Statistical image reconstruction algorithms using paraboloidal surrogates for pet transmission scans , 1999 .

[21]  J. Fessler,et al.  Maximum-likelihood transmission image reconstruction for overlapping transmission beams , 2000, IEEE Transactions on Medical Imaging.

[22]  D. Hunter,et al.  Optimization Transfer Using Surrogate Objective Functions , 2000 .

[23]  Ken D. Sauer,et al.  Parallelizable Bayesian tomography algorithms with rapid, guaranteed convergence , 2000, IEEE Trans. Image Process..

[24]  J. Bowsher,et al.  Simultaneous reconstruction and motion estimation for gated cardiac ECT , 2001, 2001 IEEE Nuclear Science Symposium Conference Record (Cat. No.01CH37310).

[25]  Arkadi Nemirovski,et al.  The Ordered Subsets Mirror Descent Optimization Method with Applications to Tomography , 2001, SIAM J. Optim..

[26]  Charles L. Byrne,et al.  Likelihood maximization for list-mode emission tomographic image reconstruction , 2001, IEEE Transactions on Medical Imaging.

[27]  Jeffrey A. Fessler,et al.  Simultaneous estimation of attenuation and activity images using optimization transfer , 2001, 2001 IEEE Nuclear Science Symposium Conference Record (Cat. No.01CH37310).

[28]  Weiguo Lu,et al.  Tomographic motion detection and correction directly in sinogram space. , 2002, Physics in medicine and biology.

[29]  Hakan Erdogan,et al.  Monotonic algorithms for transmission tomography , 2002, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[30]  Ronald J. Jaszczak,et al.  Three-dimensional motion estimation with image reconstruction for gated cardiac ECT , 2003 .

[31]  J. Fessler,et al.  Joint estimation of image and deformation parameters in motion-corrected PET , 2003, 2003 IEEE Nuclear Science Symposium. Conference Record (IEEE Cat. No.03CH37515).

[32]  Jeongtae Kim Intensity based image registration using robust similarity measure and constrained optimization: Applications for radiation therapy. , 2004 .

[33]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.