An introduction to continuous optimization for imaging

A large number of imaging problems reduce to the optimization of a cost function, with typical structural properties. The aim of this paper is to describe the state of the art in continuous optimization methods for such problems, and present the most successful approaches and their interconnections. We place particular emphasis on optimal first-order schemes that can deal with typical non-smooth and large-scale objective functions used in imaging problems. We illustrate and compare the different algorithms using classical non-smooth problems in imaging, such as denoising and deblurring. Moreover, we present applications of the algorithms to more advanced problems, such as magnetic resonance imaging, multilabel image segmentation, optical flow estimation, stereo matching, and classification.

[1]  Xavier Bresson,et al.  Bregmanized Nonlocal Regularization for Deconvolution and Sparse Reconstruction , 2010, SIAM J. Imaging Sci..

[2]  Mohamed-Jalal Fadili,et al.  Learning the Morphological Diversity , 2010, SIAM J. Imaging Sci..

[3]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[4]  A. Chambolle,et al.  A remark on accelerated block coordinate descent for computing the proximity operators of a sum of convex functions , 2015 .

[5]  Valeria Ruggiero,et al.  A Variable Metric Forward-Backward Method with Extrapolation , 2015, SIAM J. Sci. Comput..

[6]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[7]  Gene H. Golub,et al.  A Nonlinear Primal-Dual Method for Total Variation-Based Image Restoration , 1999, SIAM J. Sci. Comput..

[8]  Z. Opial.  Weak convergence of the sequence of successive approximations for nonexpansive mappings , 1967 .

[9]  L. Evans.  Measure theory and fine properties of functions , 1992 .

[10]  Arkadi Nemirovski,et al.  Lectures on modern convex optimization - analysis, algorithms, and engineering applications , 2001, MPS-SIAM series on optimization.

[11]  G. M. Korpelevich.  The extragradient method for finding saddle points and other problems , 1976 .

[12]  Yunmei Chen,et al.  Optimal Primal-Dual Methods for a Class of Saddle Point Problems , 2013, SIAM J. Optim..

[13]  Bingsheng He,et al.  On the Convergence of Primal-Dual Hybrid Gradient Algorithm , 2014, SIAM J. Imaging Sci..

[14]  Marc Teboulle,et al.  A proximal-based decomposition method for convex minimization problems , 1994, Math. Program..

[15]  Guillermo Sapiro,et al.  A Variational Framework for Exemplar-Based Image Inpainting , 2011, International Journal of Computer Vision.

[16]  Arkadi Nemirovski,et al.  Robust Convex Optimization , 1998, Math. Oper. Res..

[17]  Karen O. Egiazarian,et al.  BM3D Frames and Variational Image Deblurring , 2011, IEEE Transactions on Image Processing.

[18]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[19]  Mila Nikolova,et al.  Regularizing Flows for Constrained Matrix-Valued Images , 2004, Journal of Mathematical Imaging and Vision.

[20]  Alberto Bemporad,et al.  Douglas-rachford splitting: Complexity estimates and accelerated variants , 2014, 53rd IEEE Conference on Decision and Control.

[21]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[22]  Andrea Braides.  Approximation of Free-Discontinuity Problems , 1998 .

[23]  Adrian S. Lewis,et al.  The [barred L]ojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems , 2006, SIAM J. Optim..

[24]  Brendt Wohlberg,et al.  A nonconvex ADMM algorithm for group sparsity with sparse groups , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[25]  Mohamed-Jalal Fadili,et al.  Activity Identification and Local Linear Convergence of Douglas-Rachford/ADMM under Partial Smoothness , 2014, SSVM.

[26]  Mila Nikolova,et al.  Algorithms for Finding Global Minimizers of Image Segmentation and Denoising Models , 2006, SIAM J. Appl. Math..

[27]  Konrad Polthier,et al.  Combinatorial Image Analysis , 2014, Lecture Notes in Computer Science.

[28]  Marc Teboulle,et al.  Entropic Proximal Mappings with Applications to Nonlinear Programming , 1992, Math. Oper. Res..

[29]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[30]  Bingsheng He,et al.  On non-ergodic convergence rate of Douglas–Rachford alternating direction method of multipliers , 2014, Numerische Mathematik.

[31]  Grgoire Montavon,et al.  Neural Networks: Tricks of the Trade , 2012, Lecture Notes in Computer Science.

[32]  Daniel Cremers,et al.  The Elastic Ratio: Introducing Curvature Into Ratio-Based Image Segmentation , 2011, IEEE Transactions on Image Processing.

[33]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[34]  Antonin Chambolle,et al.  An Upwind Finite-Difference Method for Total Variation-Based Image Smoothing , 2011, SIAM J. Imaging Sci..

[35]  Stéphane Mallat,et al.  Solving Inverse Problems With Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity , 2010, IEEE Transactions on Image Processing.

[36]  James V. Burke,et al.  On the superlinear convergence of the variable metric proximal point algorithm using Broyden and BFGS matrix secant updating , 2000, Math. Program..

[37]  Laurent D. Cohen,et al.  Non-local Regularization of Inverse Problems , 2008, ECCV.

[38]  Jérôme Darbon,et al.  Image Restoration with Discrete Constrained Total Variation Part II: Levelable Functions, Convex Priors and Non-Convex Cases , 2006, Journal of Mathematical Imaging and Vision.

[39]  Daniel Cremers,et al.  A Convex Approach to Minimal Partitions , 2012, SIAM J. Imaging Sci..

[40]  Helmut Schaefer,et al.  Über die Methode sukzessiver Approximationen. , 1957 .

[41]  Francis R. Bach,et al.  Primal-dual algorithms for non-negative matrix factorization with the Kullback-Leibler divergence , 2014, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[42]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[43]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[44]  Michael Elad,et al.  Closed-Form MMSE Estimation for Signal Denoising Under Sparse Representation Modeling Over a Unitary Dictionary , 2010, IEEE Transactions on Signal Processing.

[45]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[46]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[47]  Gregory B. Passty.  Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[48]  W. R. Mann,et al.  Mean value methods in iteration , 1953 .

[49]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[50]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory, Second Edition , 2000, Statistics for Engineering and Information Science.

[51]  H. Brezis.  Analyse fonctionnelle : théorie et applications , 1983 .

[52]  R. Dykstra.  An Algorithm for Restricted Least Squares Regression , 1983 .

[53]  Daniel Cremers,et al.  The Natural Vectorial Total Variation Which Arises from Geometric Measure Theory , 2012, SIAM J. Imaging Sci..

[54]  Panagiotis Patrinos,et al.  Forward–backward quasi-Newton methods for nonsmooth optimization problems , 2016, Computational Optimization and Applications.

[55]  Pauline Tan,et al.  Acceleration of saddle-point methods in smooth cases , 2016, 1612.04141.

[56]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[57]  Jonathan M. Borwein,et al.  Duality and Convex Programming , 2015, Handbook of Mathematical Methods in Imaging.

[58]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Thomas Pock,et al.  Convex Relaxation of a Class of Vertex Penalizing Functionals , 2013, Journal of Mathematical Imaging and Vision.

[60]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[61]  Stéphane Mallat,et al.  Super-Resolution With Sparse Mixing Estimators , 2010, IEEE Transactions on Image Processing.

[62]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[63]  Gary Lawlor,et al.  PAIRED CALIBRATIONS APPLIED TO SOAP FILMS, IMMISCIBLE FLUIDS, AND SURFACES OR NETWORKS MINIMIZING OTHER NORMS , 1994 .

[64]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[65]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[66]  Jonathan Eckstein,et al.  Nonlinear Proximal Point Algorithms Using Bregman Functions, with Applications to Convex Programming , 1993, Math. Oper. Res..

[67]  Christoph Schnörr,et al.  Continuous Multiclass Labeling Approaches and Algorithms , 2011, SIAM J. Imaging Sci..

[68]  P. Davies,et al.  Local Extremes, Runs, Strings and Multiresolution , 2001 .

[69]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[70]  Tuomo Valkonen,et al.  A primal–dual hybrid gradient method for nonlinear operators with applications to MRI , 2013, 1309.5032.

[71]  Tony F. Chan,et al.  Aspects of Total Variation Regularized L[sup 1] Function Approximation , 2005, SIAM J. Appl. Math..

[72]  Daniel Cremers,et al.  An algorithm for minimizing the Mumford-Shah functional , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[73]  Patrick L. Combettes,et al.  An Algorithm for Splitting Parallel Sums of Linearly Composed Monotone Operators, with Applications to Signal Recovery , 2013, 1305.5828.

[74]  Antonin Chambolle,et al.  Partial differential equations and image processing , 1994, Proceedings of 1st International Conference on Image Processing.

[75]  Marc Teboulle,et al.  Rate of Convergence Analysis of Decomposition Methods Based on the Proximal Method of Multipliers for Convex Minimization , 2014, SIAM J. Optim..

[76]  J. Burke,et al.  A Variable Metric Proximal Point Algorithm for Monotone Operators , 1999 .

[77]  Patrick L. Combettes,et al.  A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality , 2010, SIAM J. Optim..

[78]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[79]  Zeyuan Allen Zhu,et al.  Linear Coupling: An Ultimate Unification of Gradient and Mirror Descent , 2014, ITCS.

[80]  Gabriel Peyré,et al.  Iterative Bregman Projections for Regularized Transportation Problems , 2014, SIAM J. Sci. Comput..

[81]  Kazufumi Ito,et al.  The augmented lagrangian method for equality and inequality constraints in hilbert spaces , 1990, Math. Program..

[82]  Alexandre d'Aspremont,et al.  Smooth Optimization with Approximate Gradient , 2005, SIAM J. Optim..

[83]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[84]  P. Tseng.  Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization , 2001 .

[85]  Antonin Chambolle,et al.  Image restoration by constrained total variation minimization and variants , 1995, Optics & Photonics.

[86]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[87]  Nicholas A. Johnson,et al.  A Dynamic Programming Algorithm for the Fused Lasso and L 0-Segmentation , 2013 .

[88]  Peter Richtárik,et al.  Smooth minimization of nonsmooth functions with parallel coordinate descent methods , 2013, Modeling and Optimization: Theory and Applications.

[89]  H. Brezis.  Opérateurs maximaux monotones et semi-groupes de contractions dans les espaces de Hilbert , 1973 .

[90]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[91]  Thomas Pock,et al.  A Convex, Lower Semicontinuous Approximation of Euler's Elastica Energy , 2015, SIAM J. Math. Anal..

[92]  B. Vu.  A Variable Metric Extension of the Forward–Backward–Forward Algorithm for Monotone Operators , 2012, 1210.2986.

[93]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[94]  Giovanna Citti,et al.  A Cortical Based Model of Perceptual Completion in the Roto-Translation Space , 2006, Journal of Mathematical Imaging and Vision.

[95]  G. Easley,et al.  Sparse directional image representations using the discrete shearlet transform , 2008 .

[96]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[97]  Mohamed-Jalal Fadili,et al.  Local Linear Convergence of Forward-Backward under Partial Smoothness , 2014, NIPS.

[98]  F. Browder,et al.  The solution by iteration of nonlinear functional equations in Banach spaces , 1966 .

[99]  K. Brakke.  Soap films and covering spaces , 1995 .

[100]  Donald Geman,et al.  Constrained Restoration and the Recovery of Discontinuities , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[101]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[102]  A. A. Potapenko,et al.  Method of Successive Approximations , 1964, Encyclopedia of Evolutionary Psychological Science.

[103]  Jan-Michael Frahm,et al.  Fast Global Labeling for Real-Time Stereo Using Multiple Plane Sweeps , 2008, VMV.

[104]  Marc Teboulle,et al.  Convergence Analysis of a Proximal-Like Minimization Algorithm Using Bregman Functions , 1993, SIAM J. Optim..

[105]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[106]  R. DeVore,et al.  Nonlinear approximation , 1998, Acta Numerica.

[107]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[108]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[109]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[110]  A. Chambolle.  An algorithm for Mean Curvature Motion , 2004 .

[111]  Koji Kurata,et al.  Properties of basis functions generated by shift invariant sparse representations of natural images , 2000, Biological Cybernetics.

[112]  Michael I. Jordan,et al.  A General Analysis of the Convergence of ADMM , 2015, ICML.

[113]  T. Pock,et al.  Second order total generalized variation (TGV) for MRI , 2011, Magnetic resonance in medicine.

[114]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[115]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[116]  Bingsheng He,et al.  Block-wise Alternating Direction Method of Multipliers for Multiple-block Convex Programming and Beyond , 2015 .

[117]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[118]  Harald Haas,et al.  Asilomar Conference on Signals, Systems, and Computers , 2006 .

[119]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[120]  Xiaoming Yuan,et al.  Adaptive Primal-Dual Hybrid Gradient Methods for Saddle-Point Problems , 2013, 1305.0546.

[121]  A. Moudafi,et al.  Convergence of a splitting inertial proximal method for monotone operators , 2003 .

[122]  Guido Herrmann,et al.  Adaptive neural network dynamic surface control for musculoskeletal robots , 2014, 53rd IEEE Conference on Decision and Control.

[123]  D. Hubel,et al.  Receptive fields of single neurones in the cat's striate cortex , 1959, The Journal of physiology.

[124]  Amir Beck,et al.  On the Convergence of Alternating Minimization for Convex Programming with Applications to Iteratively Reweighted Least Squares and Decomposition Schemes , 2015, SIAM J. Optim..

[125]  M. Fukushima,et al.  A minimization method for the sum of a convex function and a continuously differentiable function , 1981 .

[126]  Luca Baldassarre,et al.  Accelerated and Inexact Forward-Backward Algorithms , 2013, SIAM J. Optim..

[127]  D. Mumford,et al.  Optimal approximations by piecewise smooth functions and associated variational problems , 1989 .

[128]  Jean-Luc Starck,et al.  Deconvolution under Poisson noise using exact data fidelity and synthesis or analysis sparsity priors , 2011, 1103.2213.

[129]  Radu Ioan Bot,et al.  On the convergence rate improvement of a primal-dual splitting algorithm for solving monotone inclusion problems , 2013, Mathematical Programming.

[130]  Olivier Lezoray,et al.  Image Processing and Analysis With Graphs: theory and Practice , 2017 .

[131]  Wotao Yin,et al.  Error Forgetting of Bregman Iteration , 2013, J. Sci. Comput..

[132]  Guillermo Sapiro,et al.  Anisotropic diffusion of multivalued images with applications to color filtering , 1996, IEEE Trans. Image Process..

[133]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[134]  Gabriele Steidl,et al.  Removing Multiplicative Noise by Douglas-Rachford Splitting Methods , 2010, Journal of Mathematical Imaging and Vision.

[135]  Kristian Bredies,et al.  Preconditioned Douglas–Rachford Algorithms for TV- and TGV-Regularized Variational Imaging Problems , 2015, Journal of Mathematical Imaging and Vision.

[136]  K. Bredies,et al.  Linear Convergence of Iterative Soft-Thresholding , 2007, 0709.1598.

[137]  Daniel Cremers,et al.  Globally Optimal Image Segmentation with an Elastic Shape Prior , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[138]  Klaus Diepold,et al.  Analysis Operator Learning and its Application to Image Reconstruction , 2012, IEEE Transactions on Image Processing.

[139]  Gunnar Rätsch,et al.  Advances in Neural Information Processing Systems 21 (NIPS 2008) , 2008 .

[140]  Aharon Ben-Tal,et al.  Lectures on modern convex optimization , 1987 .

[141]  Benjamin Recht,et al.  Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints , 2014, SIAM J. Optim..

[142]  Michael Elad,et al.  The Cosparse Analysis Model and Algorithms , 2011, ArXiv.

[143]  P. Tseng.  Applications of splitting algorithm to decomposition in convex programming and variational inequalities , 1991 .

[144]  Laurent Demanet,et al.  Fast Discrete Curvelet Transforms , 2006, Multiscale Model. Simul..

[145]  Bingsheng He,et al.  On the convergence rate of Douglas–Rachford operator splitting method , 2015, Math. Program..

[146]  Mohamed-Jalal Fadili,et al.  A quasi-Newton proximal splitting method , 2012, NIPS.

[147]  Antonin Chambolle,et al.  On Total Variation Minimization and Surface Evolution Using Parametric Maximum Flows , 2009, International Journal of Computer Vision.

[148]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[149]  Wang-Q Lim,et al.  Compactly supported shearlets are optimally sparse , 2010, J. Approx. Theory.

[150]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[151]  Mark W. Schmidt,et al.  Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization , 2011, NIPS.

[152]  Antonin Chambolle,et al.  Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage , 1998, IEEE Trans. Image Process..

[153]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[154]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[155]  E. Giusti.  Minimal surfaces and functions of bounded variation , 1977 .

[156]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[157]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[158]  José M. Bioucas-Dias,et al.  A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration , 2007, IEEE Transactions on Image Processing.

[159]  Pascal Getreuer,et al.  Total Variation Deconvolution using Split Bregman , 2012, Image Process. Line.

[160]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[161]  Thomas Brox,et al.  iPiasco: Inertial Proximal Algorithm for Strongly Convex Optimization , 2015, Journal of Mathematical Imaging and Vision.

[162]  Gilbert Strang,et al.  Maximum flows and minimum cuts in the plane , 2010, J. Glob. Optim..

[163]  G. Bouchitté,et al.  The calibration method for the Mumford-Shah functional and free-discontinuity problems , 2001, math/0105013.

[164]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[165]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[166]  H. Attouch,et al.  An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping , 2001 .

[167]  Simon Setzer,et al.  Operator Splittings, Bregman Methods and Frame Shrinkage in Image Processing , 2011, International Journal of Computer Vision.

[168]  Robert E. Tarjan,et al.  A Fast Parametric Maximum Flow Algorithm and Applications , 1989, SIAM J. Comput..

[169]  Guillermo Sapiro,et al.  Supervised Dictionary Learning , 2008, NIPS.

[170]  Jérôme Darbon,et al.  Exact Optimization of Discrete Constrained Total Variation Minimization Problems , 2004, IWCIA.

[171]  P. Tseng,et al.  Block-Coordinate Gradient Descent Method for Linearly Constrained Nonsmooth Separable Optimization , 2009 .

[172]  Mohamed-Jalal Fadili,et al.  Convergence rates with inexact non-expansive operators , 2014, Mathematical Programming.

[173]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[174]  D. Bertsekas,et al.  A DESCENT NUMERICAL METHOD FOR OPTIMIZATION PROBLEMS WITH NONDIFFERENTIABLE COST FUNCTIONALS , 1973 .

[175]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[176]  Marc Teboulle,et al.  Interior Gradient and Epsilon-Subgradient Descent Methods for Constrained Convex Minimization , 2004, Math. Oper. Res..

[177]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[178]  Arkadi Nemirovski,et al.  Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems , 2004, SIAM J. Optim..

[179]  J. Gillis,et al.  Matrix Iterative Analysis , 1961 .

[180]  S. Sathiya Keerthi,et al.  Which Is the Best Multiclass SVM Method? An Empirical Study , 2005, Multiple Classifier Systems.

[181]  Dirk A. Lorenz,et al.  Minimization of Non-smooth, Non-convex Functionals by Iterative Thresholding , 2015, J. Optim. Theory Appl..

[182]  Tony F. Chan,et al.  Color TV: total variation methods for restoration of vector-valued images , 1998, IEEE Trans. Image Process..

[183]  Heinz H. Bauschke,et al.  Fixed-Point Algorithms for Inverse Problems in Science and Engineering , 2011, Springer Optimization and Its Applications.

[184]  Daniel Cremers,et al.  Convex Relaxation of Vectorial Problems with Coupled Regularization , 2014, SIAM J. Imaging Sci..

[185]  Dima Damen,et al.  Detecting Carried Objects in Short Video Sequences , 2008, ECCV.

[186]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[187]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[188]  J. Frédéric Bonnans,et al.  A family of variable metric proximal methods , 1995, Math. Program..

[189]  Joan Bruna,et al.  Signal recovery from Pooling Representations , 2013, ICML.

[190]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[191]  S. K. Zavriev,et al.  Heavy-ball method in nonconvex optimization problems , 1993 .

[192]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[193]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .

[194]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[195]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[196]  M. Fukushima,et al.  A generalized proximal point algorithm for certain non-convex minimization problems , 1981 .

[197]  K. Kiwiel.  Proximal Minimization Methods with Generalized Bregman Functions , 1997 .

[198]  Peter Richtárik,et al.  Accelerated, Parallel, and Proximal Coordinate Descent , 2013, SIAM J. Optim..

[199]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[200]  Damek Davis,et al.  Convergence Rate Analysis of Primal-Dual Splitting Schemes , 2014, SIAM J. Optim..

[201]  A. Chambolle.  FINITE-DIFFERENCES DISCRETIZATIONS OF THE MUMFORD-SHAH FUNCTIONAL , 1999 .

[202]  Daniel Cremers,et al.  Global Solutions of Variational Models with Convex Regularization , 2010, SIAM J. Imaging Sci..

[203]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[204]  Vladimir Kolmogorov,et al.  Total Variation on a Tree , 2015, SIAM J. Imaging Sci..

[205]  G. Minty.  Monotone (nonlinear) operators in Hilbert space , 1962 .

[206]  G. Naber,et al.  Encyclopedia of Mathematical Physics , 2006 .

[207]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[208]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[209]  G. D. Maso,et al.  Local calibrations for minimizers of the Mumford–Shah functional with rectilinear discontinuity sets , 2000, math/0006073.

[210]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[211]  T. Chan,et al.  Fast dual minimization of the vectorial total variation norm and applications to color image processing , 2008 .

[212]  Michael Möller,et al.  Collaborative Total Variation: A General Framework for Vectorial TV Models , 2015, SIAM J. Imaging Sci..

[213]  Antonin Chambolle,et al.  Total Variation Minimization and a Class of Binary MRF Models , 2005, EMMCVPR.

[214]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[215]  M. J. D. Powell,et al.  A method for nonlinear constraints in minimization problems , 1969 .

[216]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[217]  L. Ambrosio,et al.  Functions of Bounded Variation and Free Discontinuity Problems , 2000 .

[218]  Frédo Durand,et al.  Efficient marginal likelihood optimization in blind deconvolution , 2011, CVPR 2011.

[219]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[220]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[221]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[222]  Osman Güler,et al.  New Proximal Point Algorithms for Convex Minimization , 1992, SIAM J. Optim..

[223]  Markus Grasmair,et al.  Non-convex sparse regularisation , 2010 .

[224]  E. Sidky,et al.  Accurate image reconstruction from few-views and limited-angle data in divergent-beam CT , 2009, 0904.4495.

[225]  David Mumford,et al.  Filtering, Segmentation and Depth , 1993, Lecture Notes in Computer Science.

[226]  VekslerOlga,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001 .

[227]  Jorge Nocedal,et al.  Remark on “algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization” , 2011, TOMS.

[228]  Giorgio C. Buttazzo,et al.  Variational Analysis in Sobolev and BV Spaces - Applications to PDEs and Optimization, Second Edition , 2014, MPS-SIAM series on optimization.

[229]  Shiqian Ma,et al.  Fast Multiple-Splitting Algorithms for Convex Optimization , 2009, SIAM J. Optim..

[230]  Valeria Ruggiero,et al.  On the Convergence of Primal–Dual Hybrid Gradient Algorithms for Total Variation Image Restoration , 2012, Journal of Mathematical Imaging and Vision.

[231]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[232]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[233]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[234]  A. Auslender.  Optimisation : méthodes numériques , 1976 .

[235]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[236]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[237]  Antonin Chambolle,et al.  A l1-Unified Variational Framework for Image Restoration , 2004, ECCV.

[238]  L. Bregman.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming , 1967 .

[239]  Gilbert Strang,et al.  Maximal flow through a domain , 1983, Math. Program..

[240]  Osman Güer.  On the convergence of the proximal point algorithm for convex minimization , 1991 .

[241]  Jean-Michel Morel,et al.  Non-Local Means Denoising , 2011, Image Process. Line.

[242]  D. Donoho,et al.  Basis pursuit , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[243]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[244]  Daniel Boley,et al.  Local Linear Convergence of ISTA and FISTA on the LASSO Problem , 2015, SIAM J. Optim..

[245]  Kristian Bredies,et al.  Preconditioned Douglas-Rachford Splitting Methods for Convex-concave Saddle-point Problems , 2015, SIAM J. Numer. Anal..

[246]  M. Morini,et al.  Local calibrations for minimizers of the Mumford–Shah functional with a regular discontinuity set , 2000, math/0006032.

[247]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[248]  Daniel Cremers,et al.  Tight Convex Relaxations for Vector-Valued Labeling , 2013, SIAM J. Imaging Sci..

[249]  Valeria Ruggiero,et al.  Inexact Bregman iteration with an application to Poisson data reconstruction , 2013 .

[250]  B. Martinet.  Brève communication. Régularisation d'inéquations variationnelles par approximations successives , 1970 .

[251]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[252]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[253]  I. Yamada,et al.  Over-relaxation of the fast iterative shrinkage-thresholding algorithm with variable stepsize , 2011 .

[254]  Yunjin Chen,et al.  Insights Into Analysis Operator Learning: From Patch-Based Sparse Models to Higher Order MRFs , 2014, IEEE Transactions on Image Processing.

[255]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[256]  Thomas Brox,et al.  iPiano: Inertial Proximal Algorithm for Nonconvex Optimization , 2014, SIAM J. Imaging Sci..

[257]  Stéphane Mallat,et al.  Invariant Scattering Convolution Networks , 2012, IEEE transactions on pattern analysis and machine intelligence.

[258]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[259]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[260]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[261]  Jean-François Aujol,et al.  Stability of Over-Relaxations for the Forward-Backward Algorithm, Application to FISTA , 2015, SIAM J. Optim..

[262]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[263]  Zhaosong Lu,et al.  An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization , 2014, 1407.1296.

[264]  H. D. Ratliff,et al.  Minimum cuts and related problems , 1975, Networks.

[265]  O. Scherzer,et al.  Necessary and sufficient conditions for linear convergence of ℓ1‐regularization , 2011 .

[266]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[267]  Christophe Andrieu,et al.  31st International Conference on Machine Learning, ICML 2014 , 2014, International Conference on Machine Learning.

[268]  Mohamed-Jalal Fadili,et al.  A Generalized Forward-Backward Splitting , 2011, SIAM J. Imaging Sci..

[269]  Ravindra K. Ahuja,et al.  Network Flows: Theory, Algorithms, and Applications , 1993 .

[270]  Nelly Pustelnik,et al.  Parallel Proximal Algorithm for Image Restoration Using Hybrid Regularization , 2009, IEEE Transactions on Image Processing.

[271]  Michael Möller,et al.  The Primal-Dual Hybrid Gradient Method for Semiconvex Splittings , 2014, SIAM J. Imaging Sci..

[272]  H. Brezis,et al.  Produits infinis de resolvantes , 1978 .

[273]  Tony F. Chan,et al.  ACTIVE CONTOUR and SEGMENTATION MODELS USING GEOMETRIC PDE's for MEDICAL IMAGING ? , 2002 .

[274]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[275]  Patrick L. Combettes,et al.  A Parallel Splitting Method for Coupled Monotone Inclusions , 2009, SIAM J. Control. Optim..

[276]  Emmanuel J. Candès,et al.  Adaptive Restart for Accelerated Gradient Schemes , 2012, Foundations of Computational Mathematics.

[277]  D. Labate,et al.  Sparse Multidimensional Representations using Anisotropic Dilation and Shear Operators , 2006 .

[278]  Émilie Chouzenoux,et al.  A block coordinate variable metric forward–backward algorithm , 2016, Journal of Global Optimization.

[279]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[280]  Thomas Pock,et al.  Research data supporting 'Acceleration of the PDHGM on strongly convex subspaces'. , 2016 .

[281]  Amir Beck,et al.  On the Convergence of Block Coordinate Descent Type Methods , 2013, SIAM J. Optim..

[282]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[283]  Carlo Fischione,et al.  On the Convergence of Alternating Direction Lagrangian Methods for Nonconvex Structured Optimization Problems , 2014, IEEE Transactions on Control of Network Systems.

[284]  Daniel Cremers,et al.  A Convex Formulation of Continuous Multi-label Problems , 2008, ECCV.

[285]  Felipe Alvarez,et al.  Weak Convergence of a Relaxed and Inertial Hybrid Projection-Proximal Point Algorithm for Maximal Monotone Operators in Hilbert Space , 2003, SIAM J. Optim..

[286]  G. Bouchitté.  Convex Analysis and Duality Methods , 2006 .

[287]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[288]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[289]  Thomas Pock,et al.  Inertial Proximal Alternating Linearized Minimization (iPALM) for Nonconvex and Nonsmooth Problems , 2016, SIAM J. Imaging Sci..

[290]  Demetrio Labate,et al.  Optimally Sparse Multidimensional Representation Using Shearlets , 2007, SIAM J. Math. Anal..

[291]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[292]  Lin Xiao,et al.  An Accelerated Randomized Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization , 2015, SIAM J. Optim..

[293]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[294]  Saverio Salzo,et al.  Inexact and accelerated proximal point algorithms , 2011 .

[295]  Ernie Esser,et al.  Applications of Lagrangian-Based Alternating Direction Methods and Connections to Split Bregman , 2009 .

[296]  Émilie Chouzenoux,et al.  Variable Metric Forward–Backward Algorithm for Minimizing the Sum of a Differentiable Function and a Convex Function , 2013, Journal of Optimization Theory and Applications.

[297]  Yunmei Chen,et al.  An Accelerated Linearized Alternating Direction Method of Multipliers , 2014, SIAM J. Imaging Sci..

[298]  Julien Rabin,et al.  Regularized Discrete Optimal Transport , 2013, SIAM J. Imaging Sci..

[299]  Jérôme Darbon,et al.  Image Restoration with Discrete Constrained Total Variation Part I: Fast and Exact Optimization , 2006, Journal of Mathematical Imaging and Vision.

[300]  O. Scherzer.  Handbook of mathematical methods in imaging , 2011 .

[301]  Marc Teboulle,et al.  A simple algorithm for a class of nonsmooth convex-concave saddle-point problems , 2015, Oper. Res. Lett..

[302]  Mário A. T. Figueiredo,et al.  Deconvolving Images With Unknown Boundaries Using the Alternating Direction Method of Multipliers , 2012, IEEE Transactions on Image Processing.

[303]  Zhi-Quan Luo,et al.  Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[304]  P. Lions,et al.  Image recovery via total variation minimization and related problems , 1997 .

[305]  Dimitri P. Bertsekas,et al.  Convex Optimization Algorithms , 2015 .

[306]  A. Goldstein.  Convex programming in Hilbert space , 1964 .

[307]  Yurii Nesterov,et al.  Universal gradient methods for convex optimization problems , 2015, Math. Program..

[308]  R. Glowinski,et al.  Méthodes de Lagrangien augmenté : applications à la résolution numérique de problèmes aux limites , 1982 .

[309]  Jean-Michel Morel,et al.  A Nonlocal Bayesian Image Denoising Algorithm , 2013, SIAM J. Imaging Sci..

[310]  Gabriele Steidl,et al.  First order algorithms in variational image processing , 2014, ArXiv.

[311]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[312]  F. Browder.  Convergence theorems for sequences of nonlinear operators in Banach spaces , 1967 .

[313]  Dorit S. Hochbaum,et al.  An efficient algorithm for image segmentation, Markov random fields and related problems , 2001, JACM.

[314]  Laurent Condat,et al.  A Direct Algorithm for 1-D Total Variation Denoising , 2013, IEEE Signal Processing Letters.

[315]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[316]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[317]  Christoph Schnörr,et al.  Optimality Bounds for a Variational Relaxation of the Image Partitioning Problem , 2012, Journal of Mathematical Imaging and Vision.

[318]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[319]  Michael Möller,et al.  On the Implementation of Collaborative TV Regularization: Application to Cartoon+Texture Decomposition , 2016, Image Process. Line.

[320]  Stanley Osher,et al.  Deblurring and Denoising of Images by Nonlocal Functionals , 2005, Multiscale Model. Simul..

[321]  Ivan P. Gavrilyuk,et al.  Variational analysis in Sobolev and BV spaces , 2007, Math. Comput..

[322]  Lieven Vandenberghe,et al.  Primal-Dual Decomposition by Operator Splitting and Applications to Image Deblurring , 2014, SIAM J. Imaging Sci..

[323]  T. Hohage,et al.  A Generalization of the Chambolle-Pock Algorithm to Banach Spaces with Applications to Inverse Problems , 2014, 1412.0126.

[324]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[325]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[326]  H. Fédérer.  Geometric Measure Theory , 1969 .

[327]  Daniel Cremers,et al.  Total Cyclic Variation and Generalizations , 2013, Journal of Mathematical Imaging and Vision.

[328]  Marc Teboulle,et al.  Mirror descent and nonlinear projected subgradient methods for convex optimization , 2003, Oper. Res. Lett..

[329]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[330]  Jean-Michel Morel,et al.  On a variational theory of image amodal completion , 2006 .

[331]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[332]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[333]  A. Hero,et al.  A Fast Spectral Method for Active 3D Shape Reconstruction , 2004 .

[334]  Heinz H. Bauschke,et al.  Firmly Nonexpansive Mappings and Maximally Monotone Operators: Correspondence and Duality , 2011, 1101.4688.

[335]  Mrityunjay Kumar,et al.  Sparse Image and Signal Processing: Wavelets, Curvelets, Morphological Diversity, by Jean-Luc Starck, Fionn Murtagh, and Jalal M. Fadili , 2007 .

[336]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[337]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[338]  Alexis Bonnet,et al.  Cracktip is a global Mumford-Shah minimizer , 2018, Astérisque.

[339]  Antonin Chambolle,et al.  A Parametric Maximum Flow Approach for Discrete Total Variation Regularization , 2012 .

[340]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[341]  L. Ambrosio,et al.  A direct variational approach to a problem arising in image reconstruction , 2003 .