Block-proximal methods with spatially adapted acceleration

We study and develop (stochastic) primal--dual block-coordinate descent methods based on the method of Chambolle and Pock. Our methods have known convergence rates for the iterates and the ergodic gap: $O(1/N^2)$ if each each block is strongly convex, $O(1/N)$ if no convexity is present, and more generally a mixed rate $O(1/N^2)+O(1/N)$ for strongly convex blocks, if only some blocks are strongly convex. Additional novelties of our methods include blockwise-adapted step lengths and acceleration, as well as the ability update both the primal and dual variables randomly in blocks under a very light compatibility condition. In other words, these variants of our methods are doubly-stochastic. We test the proposed methods on various image processing problems, where we employ pixelwise-adapted acceleration.

[1]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[2]  I. Loris,et al.  On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty , 2011, 1104.1087.

[3]  Kristian Bredies,et al.  Preconditioned Douglas–Rachford Algorithms for TV- and TGV-Regularized Variational Imaging Problems , 2015, Journal of Mathematical Imaging and Vision.

[4]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[5]  Yunmei Chen,et al.  Optimal Primal-Dual Methods for a Class of Saddle Point Problems , 2013, SIAM J. Optim..

[6]  Thomas Pock,et al.  Acceleration of the PDHGM on strongly convex subspaces , 2015, ArXiv.

[7]  Wotao Yin,et al.  Bregman Iterative Algorithms for (cid:2) 1 -Minimization with Applications to Compressed Sensing ∗ , 2008 .

[8]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[9]  Min Li,et al.  Adaptive Primal-Dual Splitting Methods for Statistical Learning and Image Processing , 2015, NIPS.

[10]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[11]  Tong Zhang,et al.  Stochastic Optimization with Importance Sampling , 2014, ArXiv.

[12]  Adams Wei Yu,et al.  Doubly Stochastic Primal-Dual Coordinate Method for Empirical Risk Minimization and Bilinear Saddle-Point Problem , 2015 .

[13]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[14]  Patrick L. Combettes,et al.  Stochastic forward-backward and primal-dual approximation algorithms with application to online image restoration , 2016, 2016 24th European Signal Processing Conference (EUSIPCO).

[15]  David Stutz.  IPIANO : INERTIAL PROXIMAL ALGORITHM FOR NON-CONVEX OPTIMIZATION , 2016 .

[16]  Taiji Suzuki.  Stochastic Dual Coordinate Ascent with Alternating Direction Multiplier Method , 2013, 1311.0622.

[17]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[18]  Thomas Brox,et al.  iPiano: Inertial Proximal Algorithm for Nonconvex Optimization , 2014, SIAM J. Imaging Sci..

[19]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[20]  Bingsheng He,et al.  On the Convergence of Primal-Dual Hybrid Gradient Algorithm , 2014, SIAM J. Imaging Sci..

[21]  Peter Richtárik,et al.  Distributed Coordinate Descent Method for Learning with Big Data , 2013, J. Mach. Learn. Res..

[22]  Ming Yan,et al.  ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates , 2015, SIAM J. Sci. Comput..

[23]  Adrian S. Lewis,et al.  Partial Smoothness, Tilt Stability, and Generalized Hessians , 2013, SIAM J. Optim..

[24]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[25]  Peter Richtárik,et al.  Optimization in High Dimensions via Accelerated, Parallel, and Proximal Coordinate Descent , 2016, SIAM Rev..

[26]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[27]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[28]  Peter Richtárik,et al.  Randomized Dual Coordinate Ascent with Arbitrary Sampling , 2014, ArXiv.

[29]  Pascal Bianchi,et al.  A stochastic coordinate descent primal-dual algorithm and applications , 2014, 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP).

[30]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[31]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[32]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[33]  Peter Richtárik,et al.  Accelerated, Parallel, and Proximal Coordinate Descent , 2013, SIAM J. Optim..

[34]  Yuchen Zhang,et al.  Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimization , 2014, ICML.

[35]  T. Hohage,et al.  A Generalization of the Chambolle-Pock Algorithm to Banach Spaces with Applications to Inverse Problems , 2014, 1412.0126.

[36]  J. Pesquet,et al.  A Class of Randomized Primal-Dual Algorithms for Distributed Optimization , 2014, 1406.6404.

[37]  Peter Richtárik,et al.  Parallel coordinate descent methods for big data optimization , 2012, Mathematical Programming.

[38]  Simon Setzer,et al.  Operator Splittings, Bregman Methods and Frame Shrinkage in Image Processing , 2011, International Journal of Computer Vision.

[39]  Michael Möller,et al.  The Primal-Dual Hybrid Gradient Method for Semiconvex Splittings , 2014, SIAM J. Imaging Sci..

[40]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[41]  Dimitri P. Bertsekas,et al.  Incremental Aggregated Proximal and Augmented Lagrangian Algorithms , 2015, ArXiv.

[42]  Tong Zhang,et al.  Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization , 2013, Mathematical Programming.

[43]  Ming Yan,et al.  Coordinate Friendly Structures, Algorithms and Applications , 2016, ArXiv.

[44]  K. Bredies,et al.  Total generalised variation in diffusion tensor imaging , 2012 .

[45]  Mohamed-Jalal Fadili,et al.  Local Linear Convergence of Forward-Backward under Partial Smoothness , 2014, NIPS.

[46]  Bingsheng He,et al.  Block-wise Alternating Direction Method of Multipliers for Multiple-block Convex Programming and Beyond , 2015 .

[47]  Peter Richtárik,et al.  Stochastic Dual Coordinate Ascent with Adaptive Probabilities , 2015, ICML.

[48]  Tianbao Yang,et al.  Doubly Stochastic Primal-Dual Coordinate Method for Bilinear Saddle-Point Problem , 2015, 1508.03390.

[49]  Adrian S. Lewis,et al.  Active Sets, Nonsmoothness, and Sensitivity , 2002, SIAM J. Optim..

[50]  Suzuki Taiji,et al.  Stochastic Dual Coordinate Ascent with Alternating Direction Multiplier Method , 2013 .

[51]  Carola-Bibiane Schönlieb,et al.  Bilevel Parameter Learning for Higher-Order Total Variation Regularisation Models , 2015, Journal of Mathematical Imaging and Vision.

[52]  J. Koenderink.  Q… , 2014, Les noms officiels des communes de Wallonie, de Bruxelles-Capitale et de la communaute germanophone.

[53]  P. Lions,et al.  Image recovery via total variation minimization and related problems , 1997 .

[54]  Daniel Cremers,et al.  An algorithm for minimizing the Mumford-Shah functional , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[55]  Pascal Bianchi,et al.  A Coordinate Descent Primal-Dual Algorithm and Application to Distributed Asynchronous Optimization , 2014, IEEE Transactions on Automatic Control.

[56]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[57]  Stephen J. Wright.  Coordinate descent algorithms , 2015, Mathematical Programming.

[58]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[59]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[60]  A. Chambolle.  An algorithm for Mean Curvature Motion , 2004 .

[61]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[62]  Tuomo Valkonen,et al.  Testing and Non-linear Preconditioning of the Proximal Point Method , 2017, Applied Mathematics & Optimization.

[63]  Peter Richtárik,et al.  SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization , 2015, ICML.

[64]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .