Compressive Algorithms-Adaptive Solutions of PDEs and Variational Problems

This paper is concerned with an overview of the main concepts and a few significant applications of a class of adaptive iterative algorithms which allow for dimensionality reductions when used to solve large scale problems. We call this class of numerical methods Compressive Algorithms . The introduction of this paper presents an historical excursus on the developments of the main ideas behind compressive algorithms and stresses the common features of diverse applications. The first part of the paper is addressed to the optimal performances of such algorithms when compared with known benchmarks in the numerical solution of elliptic partial differential equations. In the second part we address the solution of inverse problems both with sparsity and compressibility constraints. We stress how compressive algorithms can stem from variational principles. We illustrate the main results and applications by a few significant numerical examples. We conclude by pointing out future developments.

[1]  M. Fornasier,et al.  Iterative thresholding algorithms , 2008 .

[2]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[3]  L. Elsgolts.  Differential Equations and the Calculus of Variations , 2003 .

[4]  Thorsten Raasch,et al.  Adaptive Wavelet and Frame Schemes for Elliptic and Parabolic Equations , 2007 .

[5]  H. L. Taylor,et al.  Deconvolution with the l 1 norm , 1979 .

[6]  Massimo Fornasier,et al.  Iterative Thresholding Meets Free-Discontinuity Problems , 2009, Found. Comput. Math..

[7]  M. Fornasier,et al.  Adaptive Frame Methods for Elliptic Operator Equations: The Steepest Descent Approach , 2007 .

[8]  Bruno Torrésani,et al.  Hybrid representations for audiophonic signal encoding , 2002, Signal Process..

[9]  Gjlles Aubert,et al.  Mathematical problems in image processing , 2001 .

[10]  L. Vese.  A Study in the BV Space of a Denoising—Deblurring Variational Problem , 2001 .

[11]  David L. Donoho,et al.  High-Dimensional Centrally Symmetric Polytopes with Neighborliness Proportional to Dimension , 2006, Discret. Comput. Geom..

[12]  Carola-Bibiane Schönlieb,et al.  Subspace Correction Methods for Total Variation and 1-Minimization , 2007, SIAM J. Numer. Anal..

[13]  Claudio Canuto,et al.  The wavelet element method. Part I: Construction and analysis. , 1997 .

[14]  G. Teschke.  Multi-frame representations in linear inverse problems with mixed multi-constraints , 2007 .

[15]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[16]  I. Daubechies,et al.  Iteratively solving linear inverse problems under general convex constraints , 2007 .

[17]  Rob P. Stevenson,et al.  A multiplicative Schwarz adaptive wavelet method for elliptic boundary value problems , 2009, Math. Comput..

[18]  I. Daubechiesa,et al.  Variational image restoration by means of wavelets : Simultaneous decomposition , deblurring , and denoising , 2005 .

[19]  I. Daubechies,et al.  Wavelets on the Interval and Fast Wavelet Transforms , 1993 .

[20]  Wolfgang Dahmen,et al.  Adaptive Wavelet Schemes for Nonlinear Variational Problems , 2003, SIAM J. Numer. Anal..

[21]  P. Lions,et al.  Image recovery via total variation minimization and related problems , 1997 .

[22]  J. Claerbout,et al.  Robust Modeling With Erratic Data , 1973 .

[23]  Jinchao Xu,et al.  Iterative Methods by Space Decomposition and Subspace Correction , 1992, SIAM Rev..

[24]  D. Donoho.  Superresolution via sparsity constraints , 1992 .

[25]  Antonin Chambolle,et al.  Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage , 1998, IEEE Trans. Image Process..

[26]  Wolfgang Dahmen,et al.  Local Decomposition of Refinable Spaces and Wavelets , 1996 .

[27]  Wolfgang Dahmen,et al.  Stable multiscale bases and local error estimation for elliptic problems , 1997 .

[28]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[29]  M. Fornasier.  Quasi-orthogonal decompositions of structured frames , 2004 .

[30]  Wolfgang Dahmen,et al.  Adaptive Finite Element Methods with convergence rates , 2004, Numerische Mathematik.

[31]  P. Grisvard.  Elliptic Problems in Nonsmooth Domains , 1985 .

[32]  I. Loris.  On the performance of algorithms for the minimization of ℓ1-penalized functionals , 2007, 0710.4082.

[33]  R. DeVore,et al.  Nonlinear approximation , 1998, Acta Numerica.

[34]  R. Duffin,et al.  A class of nonharmonic Fourier series , 1952 .

[35]  Peter Oswald.  Multilevel Frames and Riesz Bases in Sobolev Spaces , 2004 .

[36]  Massimo Fornasier,et al.  Adaptive frame methods for nonlinear variational problems , 2008, Numerische Mathematik.

[37]  G. Constantine,et al.  A Multivariate Faa di Bruno Formula with Applications , 1996 .

[38]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[39]  Angela Kunoth,et al.  Wavelets on manifolds: An optimized construction , 2006, Math. Comput..

[40]  D. Mumford,et al.  Optimal approximations by piecewise smooth functions and associated variational problems , 1989 .

[41]  W. Dahmen,et al.  Biorthogonal Spline Wavelets on the Interval—Stability and Moment Conditions , 1999 .

[42]  R. Gribonval,et al.  Highly sparse representations from dictionaries are unique and independent of the sparseness measure , 2007 .

[43]  Reinhard H Stephan Dahlke.  Adaptive Wavelet Methods for Saddle Point Problems , 1999 .

[44]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[45]  Rob P. Stevenson,et al.  Wavelets with patchwise cancellation properties , 2006, Math. Comput..

[46]  E. Candès,et al.  Astronomical image representation by the curvelet transform , 2003, Astronomy & Astrophysics.

[47]  Wolfgang Dahmen,et al.  Adaptive wavelet methods for elliptic operator equations: Convergence rates , 2001, Math. Comput..

[48]  Antonin Chambolle,et al.  Implementation of an adaptive finite-element approximation of the Mumford-Shah functional , 2000, Numerische Mathematik.

[49]  M. Fornasier.  Domain decomposition methods for linear inverse problems with sparsity constraints , 2007 .

[50]  Gilles Aubert,et al.  Efficient Schemes for Total Variation Minimization Under Constraints in Image Processing , 2009, SIAM J. Sci. Comput..

[51]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[52]  O. Christensen.  An introduction to frames and Riesz bases , 2002 .

[53]  Rob P. Stevenson,et al.  Optimality of a Standard Adaptive Finite Element Method , 2007, Found. Comput. Math..

[54]  Rob P. Stevenson,et al.  Adaptive Solution of Operator Equations Using Wavelet Frames , 2003, SIAM J. Numer. Anal..

[55]  P. Laguna,et al.  Signal Processing , 2002, Yearbook of Medical Informatics.

[56]  G. D. Maso,et al.  An Introduction to-convergence , 1993 .

[57]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[58]  I. Johnstone,et al.  Wavelet Shrinkage: Asymptopia? , 1995 .

[59]  L. Schumaker.  Spline Functions: Basic Theory , 1981 .

[60]  Massimo Fornasier,et al.  The application of joint sparsity and total variation minimization algorithms to a real-life art restoration problem , 2009, Adv. Comput. Math..

[61]  Ricardo H. Nochetto,et al.  Convergence of Adaptive Finite Element Methods , 2002, SIAM Rev..

[62]  Wolfgang Dahmen,et al.  Adaptive Wavelet Methods for Saddle Point Problems - Optimal Convergence Rates , 2002, SIAM J. Numer. Anal..

[63]  Albert Cohen,et al.  Adaptive Wavelet Galerkin Methods for Linear Inverse Problems , 2004, SIAM J. Numer. Anal..

[64]  Massimo Fornasier,et al.  Domain decomposition methods for compressed sensing , 2009, 0902.0124.

[65]  Wotao Yin,et al.  An Iterative Regularization Method for Total Variation-Based Image Restoration , 2005, Multiscale Model. Simul..

[66]  I. Daubechies,et al.  Accelerated Projected Gradient Method for Linear Inverse Problems with Sparsity Constraints , 2007, 0706.4297.

[67]  C. Chui,et al.  Wavelets on a Bounded Interval , 1992 .

[68]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[69]  Wolfgang Dahmen,et al.  Nonlinear Approximation and Adaptive Techniques for Solving Elliptic Operator Equations , 1997 .

[70]  Albert Cohen,et al.  Wavelet adaptive method for second order elliptic problems: boundary conditions and domain decomposition , 2000, Numerische Mathematik.

[71]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[72]  Wolfgang Dahmen,et al.  Wavelets on Manifolds I: Construction and Domain Decomposition , 1999, SIAM J. Math. Anal..

[73]  Yuesheng Xu,et al.  Adaptive Wavelet Methods for Elliptic Operator Equations with Nonlinear Terms , 2003, Adv. Comput. Math..

[74]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[75]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[76]  L. Ambrosio,et al.  Functions of Bounded Variation and Free Discontinuity Problems , 2000 .

[77]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[78]  Antonin Chambolle,et al.  Image Segmentation by Variational Methods: Mumford and Shah Functional and the Discrete Approximations , 1995, SIAM J. Appl. Math..

[79]  Rachel Ward,et al.  On the Complexity of Mumford–Shah-Type Regularization, Viewed as a Relaxed Sparsity Constraint , 2010, IEEE Transactions on Image Processing.

[80]  Qingtang Jiang,et al.  Spectral Analysis of the Transition Operator and Its Applications to Smoothness Analysis of Wavelets , 2002, SIAM J. Matrix Anal. Appl..

[81]  D. Donoho,et al.  Uncertainty principles and signal recovery , 1989 .

[82]  Massimo Fornasier,et al.  Adaptive frame methods for elliptic operator equations , 2007, Adv. Comput. Math..

[83]  Fionn Murtagh,et al.  Fast communication , 2002 .

[84]  R. DeVore,et al.  Besov regularity for elliptic boundary value problems , 1997 .

[85]  A. Chambolle.  Practical, Unified, Motion and Missing Data Treatment in Degraded Video , 2004, Journal of Mathematical Imaging and Vision.

[86]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[87]  D. Donoho.  Nonlinear Solution of Linear Inverse Problems by Wavelet–Vaguelette Decomposition , 1995 .

[88]  Massimo Fornasier,et al.  Recovery Algorithms for Vector-Valued Data with Joint Sparsity Constraints , 2008, SIAM J. Numer. Anal..

[89]  D. Donoho,et al.  Sparse nonnegative solution of underdetermined linear equations by linear programming. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[90]  I. Daubechies,et al.  Tomographic inversion using L1-norm regularization of wavelet coefficients , 2006, physics/0608094.

[91]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[92]  R. Richardson.  The International Congress of Mathematicians , 1932, Science.

[93]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[94]  Wolfgang Dahmen,et al.  Adaptive Wavelet Schemes for Elliptic Problems - Implementation and Numerical Experiments , 2001, SIAM J. Sci. Comput..

[95]  Ronald F. Gariepy.  FUNCTIONS OF BOUNDED VARIATION AND FREE DISCONTINUITY PROBLEMS (Oxford Mathematical Monographs) , 2001 .

[96]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[97]  Wolfgang Dahmen,et al.  Multiscale Wavelet Methods for Partial Differential Equations , 1997 .

[98]  A. Gilbert,et al.  Theoretical and experimental analysis of a randomized algorithm for Sparse Fourier transform analysis , 2004, math/0411102.

[99]  I. Daubechies,et al.  Biorthogonal bases of compactly supported wavelets , 1992 .

[100]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[101]  Wolfgang Dahmen,et al.  Adaptive Wavelet Methods II—Beyond the Elliptic Case , 2002, Found. Comput. Math..

[102]  B. Logan,et al.  Signal recovery and the large sieve , 1992 .

[103]  Peter Oswald.  Frames and Space Splittings in Hilbert Spaces , 2004 .

[104]  Wolfgang Dahmen,et al.  Fast computation tools for adaptive wavelet schemes , 2005 .

[105]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[106]  Albert Cohen,et al.  Wavelet methods in numerical analysis , 2000 .

[107]  Pierre Kornprobst,et al.  Mathematical problems in image processing - partial differential equations and the calculus of variations , 2010, Applied mathematical sciences.

[108]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[109]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[110]  Rob Stevenson,et al.  Computation of differential operators in aggregated wavelet frame coordinates , 2007 .

[111]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[112]  Stephan Dahlke,et al.  Besov regularity for elliptic boundary value problems in polygonal domains , 1999 .

[113]  Massimo Fornasier,et al.  Optimal adaptive computations in the Jaffard algebra and localized frames , 2010, J. Approx. Theory.

[114]  R. Vershynin,et al.  One sketch for all: fast algorithms for compressed sensing , 2007, STOC '07.

[115]  W. Dahmen,et al.  Wavelets with Complementary Boundary Conditions — Function Spaces on the Cube , 1998 .

[116]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[117]  Wolfgang Dahmen,et al.  Composite wavelet bases for operator equations , 1999, Math. Comput..

[118]  R. Gribonval,et al.  Bi-framelet systems with few vanishing moments characterize Besov spaces , 2004 .

[119]  R. LeVeque,et al.  MIT Spring 2009 { Course Information 18.336 { Numerical Methods for Partial Dierential Equations , 2009 .

[120]  Rob P. Stevenson,et al.  An Optimal Adaptive Finite Element Method , 2004, SIAM J. Numer. Anal..

[121]  A. Chambolle,et al.  Discrete approximation of the Mumford-Shah functional in dimension two , 1999, ESAIM: Mathematical Modelling and Numerical Analysis.

[122]  D. Donoho,et al.  Counting faces of randomly-projected polytopes when the projection radically lowers dimension , 2006, math/0607364.

[123]  F. Santosa,et al.  Linear inversion of ban limit reflection seismograms , 1986 .

[124]  M. Fornasier,et al.  Nonlinear and adaptive frame approximation schemes for elliptic PDEs: Theory and numerical experiments , 2009 .