Faster Gradient Descent and the Efficient Recovery of Images

Much recent attention has been devoted to gradient descent algorithms where the steepest descent step size is replaced by a similar one from a previous iteration or gets updated only once every second step, thus forming a faster gradient descent method. For unconstrained convex quadratic optimization these methods can converge much faster than steepest descent. But the context of interest here is application to certain ill-posed inverse problems, where the steepest descent method is known to have a smoothing, regularizing effect, and where a strict optimization solution is not necessary.Specifically, in this paper we examine the effect of replacing steepest descent by a faster gradient descent algorithm in the practical context of image deblurring and denoising tasks. We also propose several highly efficient schemes for carrying out these tasks independently of the step size selection, as well as a scheme for the case where both blur and significant noise are present.In the above context there are situations where many steepest descent steps are required, thus building slowness into the solution procedure. Our general conclusion regarding gradient descent methods is that in such cases the faster gradient descent methods offer substantial advantages. In other situations where no such slowness buildup arises the steepest descent method can still be very effective.

[1]  G. Sapiro,et al.  Geometric partial differential equations and image analysis [Book Reviews] , 2001, IEEE Transactions on Medical Imaging.

[2]  Suresh B. Srinivasamurthy Methods of Solving Ill-Posed Problems , 2012, 1205.5323.

[3]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[4]  Roger Fletcher,et al.  Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming , 2005, Numerische Mathematik.

[5]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  C. Vogel Computational Methods for Inverse Problems , 1987 .

[7]  Lothar Reichel,et al.  Lanczos-Based Exponential Filtering for Discrete Ill-Posed Problems , 2002, Numerical Algorithms.

[8]  Jorge Nocedal,et al.  On the Behavior of the Gradient Norm in the Steepest Descent Method , 2002, Comput. Optim. Appl..

[9]  Uri M. Ascher,et al.  The Chaotic Nature of Faster Gradient Descent Methods , 2012, J. Sci. Comput..

[10]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[11]  U. Ascher,et al.  Artificial time integration , 2007 .

[12]  Stéphane Mallat,et al.  A Wavelet Tour of Signal Processing - The Sparse Way, 3rd Edition , 2008 .

[13]  Alexei A. Efros,et al.  Fast bilateral filtering for the display of high-dynamic-range images , 2002 .

[14]  J. Hardy,et al.  Adaptive Optics for Astronomical Telescopes , 1998 .

[15]  U. Ascher,et al.  Esaim: Mathematical Modelling and Numerical Analysis Gradient Descent and Fast Artificial Time Integration , 2022 .

[16]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[17]  Ronald Fedkiw,et al.  Level set methods and dynamic implicit surfaces , 2002, Applied mathematical sciences.

[18]  T. Chan,et al.  On the Convergence of the Lagged Diffusivity Fixed Point Method in Total Variation Image Restoration , 1999 .

[19]  H. Akaike On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method , 1959 .

[20]  Guillermo Sapiro,et al.  Robust anisotropic diffusion , 1998, IEEE Trans. Image Process..

[21]  T. Hughes,et al.  Signals and systems , 2006, Genome Biology.

[22]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[23]  J. M. Martínez,et al.  Gradient Method with Retards and Generalizations , 1998 .

[24]  Tony F. Chan,et al.  Image processing and analysis - variational, PDE, wavelet, and stochastic methods , 2005 .

[25]  Marcos Raydan,et al.  Relaxed Steepest Descent and Cauchy-Barzilai-Borwein Method , 2002, Comput. Optim. Appl..

[26]  N. Gol’dman,et al.  Regularization of Ill-Posed Problems by Iteration Methods , 1999 .

[27]  D. Oldenburg,et al.  NON-LINEAR INVERSION USING GENERAL MEASURES OF DATA MISFIT AND MODEL STRUCTURE , 1998 .

[28]  J.S.Chitode Discrete Time Signal Processing , 2011 .

[29]  Hui Huang,et al.  On Effective Methods for Implicit Piecewise Smooth Surface Recovery , 2006, SIAM J. Sci. Comput..

[30]  J. Claerbout,et al.  Robust Modeling With Erratic Data , 1973 .

[31]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..