Convergence Rate Analysis of the Majorize–Minimize Subspace Algorithm

State-of-the-art methods for solving smooth optimization problems are nonlinear conjugate gradient, low memory BFGS, and majorize-minimize (MM) subspace algorithms. The MM subspace algorithm that has been introduced more recently has shown good practical performance when compared with other methods on various optimization problems arising in signal and image processing. However, to the best of our knowledge, no general result exists concerning the theoretical convergence rate of the MM subspace algorithm. This paper aims at deriving such convergence rates both for batch and online versions of the and in particular, discusses the influence of the choice of the subspace.

[1]  Robert D. Nowak,et al.  Majorization–Minimization Algorithms for Wavelet-Based Image Restoration , 2007, IEEE Transactions on Image Processing.

[2]  Prabhu Babu,et al.  Sparse Generalized Eigenvalue Problem Via Smooth Optimization , 2014, IEEE Transactions on Signal Processing.

[3]  W. Hager,et al.  A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS , 2005 .

[4]  D. Hunter,et al.  A Tutorial on MM Algorithms , 2004 .

[5]  Jeffrey A. Fessler,et al.  A paraboloidal surrogates algorithm for convergent penalized-likelihood emission image reconstruction , 1998, 1998 IEEE Nuclear Science Symposium Conference Record. 1998 IEEE Nuclear Science Symposium and Medical Imaging Conference (Cat. No.98CH36255).

[6]  Yves Goussard,et al.  On global and local convergence of half-quadratic algorithms , 2006, IEEE Transactions on Image Processing.

[7]  Émilie Chouzenoux,et al.  A Majorize–Minimize Strategy for Subspace Optimization Applied to Image Restoration , 2011, IEEE Transactions on Image Processing.

[8]  Nicholas I. M. Gould,et al.  On Iterated-Subspace Minimization Methods for Nonlinear Optimization , 1996 .

[9]  I. Selesnick,et al.  Chromatogram baseline estimation and denoising using sparsity (BEADS) , 2014 .

[10]  A. Miele,et al.  Study on a memory gradient method for the minimization of functions , 1969 .

[11]  Elijah Polak,et al.  Optimization: Algorithms and Consistent Approximations , 1997 .

[12]  Michel Barlaud,et al.  Deterministic edge-preserving regularization in computed imaging , 1997, IEEE Trans. Image Process..

[13]  Jean-Christophe Pesquet,et al.  Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed ${\ell _1}/{\ell _2}$ Regularization , 2014, IEEE Signal Processing Letters.

[14]  Émilie Chouzenoux,et al.  A Stochastic Majorize-Minimize Subspace Algorithm for Online Penalized Least Squares Estimation , 2015, IEEE Transactions on Signal Processing.

[15]  Hugues Talbot,et al.  A Memory Gradient algorithm for ℓ2 — ℓ0 regularization with applications to image restoration , 2011, 2011 18th IEEE International Conference on Image Processing.

[16]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[17]  J. Idier,et al.  Convergence of Conjugate Gradient Methods with a Closed-Form Stepsize Formula , 2008 .

[18]  K. Lange.  A gradient algorithm locally equivalent to the EM algorithm , 1995 .

[19]  Zhi-Quan Luo,et al.  A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing , 2015, IEEE Signal Processing Magazine.

[20]  Jérôme Idier,et al.  Convex half-quadratic criteria and interacting auxiliary variables for image restoration , 2001, IEEE Trans. Image Process..

[21]  A. Ostrowski.  Solution of equations in Euclidean and Banach spaces , 1973 .

[22]  Émilie Chouzenoux,et al.  A Majorize-Minimize Memory Gradient method for complex-valued inverse problems , 2014, Signal Process..

[23]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[24]  Zhihua Zhang,et al.  Surrogate maximization/minimization algorithms and extensions , 2007, Machine Learning.

[25]  Jeffrey A. Fessler,et al.  An Expanded Theoretical Treatment of Iteration-Dependent Majorize-Minimize Algorithms , 2007, IEEE Transactions on Image Processing.

[26]  Hugues Talbot,et al.  A Majorize-Minimize Subspace Approach for ℓ2-ℓ0 Image Regularization , 2011, SIAM J. Imaging Sci..

[27]  Michael Elad,et al.  Coordinate and subspace optimization methods for linear least squares with non-quadratic regularization , 2007 .

[28]  Mila Nikolova,et al.  Analysis of Half-Quadratic Minimization Methods for Signal and Image Recovery , 2005, SIAM J. Sci. Comput..

[29]  Ya-Xiang Yuan,et al.  Subspace Techniques for Nonlinear Optimization , 2007 .