The lost honor of ℓ2-based regularization

In the past two decades, regularization methods based on the `1 norm, including sparse wavelet representations and total variation, have become immensely popular. So much so, that we were led to consider the question whether `1-based techniques ought to altogether replace the simpler, faster and better known `2-based alternatives as the default approach to regularization techniques. The occasionally tremendous advances of `1-based techniques are not in doubt. However, such techniques also have their limitations. This article explores advantages and disadvantages compared to `2-based techniques using several practical case studies. Taking into account the considerable added hardship in calculating solutions of the resulting computational problems, `1-based techniques must offer substantial advantages to be worthwhile. In this light our results suggest that in many applications, though not all, `2-based recovery may still be preferred.

[1]  H. Helmholtz.  Ueber einige Gesetze der Vertheilung elektrischer Ströme in körperlichen Leitern mit Anwendung auf die thierisch‐elektrischen Versuche , 1853 .

[2]  H. M. Wagner.  Linear Programming Techniques for Regression Analysis , 1959 .

[3]  J. Claerbout,et al.  Robust Modeling With Erratic Data , 1973 .

[4]  Simon French,et al.  Finite Algorithms in Optimization and Data Analysis , 1986 .

[5]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[6]  R. Greenblatt.  Probabilistic reconstruction of multiple sources in the bioelectromagnetic inverse problem , 1993 .

[7]  M. Fuchs,et al.  Depth normalization in MEG/EEG current density imaging , 1996, Proceedings of 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[8]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[9]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[10]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[11]  G. Sapiro,et al.  Geometric partial differential equations and image analysis [Book Reviews] , 2001, IEEE Transactions on Medical Imaging.

[12]  Ronald Fedkiw,et al.  Level set methods and dynamic implicit surfaces , 2002, Applied mathematical sciences.

[13]  M. Murray,et al.  EEG source imaging , 2004, Clinical Neurophysiology.

[14]  E. Haber,et al.  Inversion of 3D electromagnetic data in frequency and time domain using an inexact all-at-once approach , 2004 .

[15]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[16]  D. Donoho,et al.  Sparse nonnegative solution of underdetermined linear equations by linear programming. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[17]  Tony F. Chan,et al.  Image processing and analysis - variational, PDE, wavelet, and stochastic methods , 2005 .

[18]  D. Donoho.  For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[19]  Hui Huang,et al.  On Effective Methods for Implicit Piecewise Smooth Surface Recovery , 2006, SIAM J. Sci. Comput..

[20]  Faming Liang,et al.  Statistical and Computational Inverse Problems , 2006, Technometrics.

[21]  R. Ilmoniemi,et al.  Interpreting magnetic fields of the brain: minimum norm estimates , 2006, Medical and Biological Engineering and Computing.

[22]  Jianhong Shen,et al.  Deblurring images: Matrices, spectra, and filtering , 2007, Math. Comput..

[23]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[24]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[25]  Daniel Cohen-Or,et al.  Parameterization-free projection for geometry reconstruction , 2007, ACM Trans. Graph..

[26]  U. Ascher,et al.  Dynamic level set regularization for large distributed parameter estimation problems , 2007 .

[27]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[28]  E. Haber,et al.  Adaptive finite volume method for distributed non-smooth parameter identification , 2007 .

[29]  R. DeVore,et al.  Compressed sensing and best k-term approximation , 2008 .

[30]  D. Pai,et al.  Computed myography : three-dimensional reconstruction of motor functions from surface EMG data , 2008 .

[31]  Dianne P. O'Leary,et al.  Deblurring Images: Matrices, Spectra and Filtering , 2006, J. Electronic Imaging.

[32]  Hui Huang,et al.  Efficient reconstruction of 2D images and 3D surfaces , 2008 .

[33]  Stéphane Mallat,et al.  A Wavelet Tour of Signal Processing - The Sparse Way, 3rd Edition , 2008 .

[34]  N. Meinshausen,et al.  Stability selection , 2008, 0809.2932.

[35]  Rayan Saab,et al.  Stable sparse approximations via nonconvex optimization , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[36]  Rick Chartrand,et al.  Fast algorithms for nonconvex compressive sensing: MRI reconstruction from very few data , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[37]  Jian-Feng Cai,et al.  Split Bregman Methods and Frame Based Image Restoration , 2009, Multiscale Model. Simul..

[38]  Daniel Cohen-Or,et al.  Consolidation of unorganized point clouds for surface reconstruction , 2009, ACM Trans. Graph..

[39]  Daniel Cohen-Or,et al.  ℓ1-Sparse reconstruction of sharp point set surfaces , 2010, TOGS.

[40]  A. Morelli.  Inverse Problem Theory , 2010 .

[41]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[42]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[43]  Panos M. Pardalos,et al.  Convex optimization theory , 2010, Optim. Methods Softw..

[44]  Arkadi Nemirovski,et al.  On verifiable sufficient conditions for sparse signal recovery via ℓ1 minimization , 2008, Math. Program..

[45]  Nando de Freitas,et al.  On Autoencoders and Score Matching for Energy Based Models , 2011, ICML.

[46]  U. Ascher,et al.  Adaptive and stochastic algorithms for EIT and DC resistivity problems with piecewise constant solutions and many measurements , 2011 .

[47]  Dinesh K. Pai,et al.  Source localization in electromyography using the inverse potential problem , 2011 .

[48]  Michael P. Friedlander,et al.  Sparse Optimization with Least-Squares Constraints , 2011, SIAM J. Optim..

[49]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[50]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[51]  Gilad Lerman,et al.  Sparse Regularization for Precipitation Downscaling , 2012 .

[52]  Lior Horesh,et al.  Optimal estimation of L1 regularization prior from a regularized empirical bayesian risk standpoint , 2012 .

[53]  Nancy Nichols,et al.  Resolution of sharp fronts in the presence of model error in variational data assimilation , 2013 .

[54]  Daniel Cohen-Or,et al.  Edge-aware point set resampling , 2013, ACM Trans. Graph..

[55]  Uri M. Ascher,et al.  Stochastic Algorithms for Inverse Problems Involving PDEs and many Measurements , 2014, SIAM J. Sci. Comput..