Customized proximal point algorithms for linearly constrained convex minimization and saddle-point problems : a uniform approach

This paper focuses on some customized applications of the proximal point algorithm (PPA) to two classes of problems: the convex minimization problem with linear constraints and a generic or separable objective function, and a saddle-point problem. We treat these two classes of problems uniformly by a mixed variational inequality, and show how the application of PPA with customized metric proximal parameters can yield favorable algorithms which are able to make use of the models’ structures effectively. Our customized PPA revisit turns out to unify some algorithms including some existing ones in the literature and some new ones to be proposed. From the PPA perspective, we establish the global convergence and a worst-case O(1/t) convergence rate for this series of algorithms in a unified way.

[1]  John Wright,et al.  RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[3]  Xiaoming Yuan,et al.  Recovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations , 2011, SIAM J. Optim..

[4]  Xiaoming Yuan,et al.  A splitting method for separable convex programming , 2015 .

[5]  Osman Güler,et al.  New Proximal Point Algorithms for Convex Minimization , 1992, SIAM J. Optim..

[6]  Gene H. Golub,et al.  A Nonlinear Primal-Dual Method for Total Variation-Based Image Restoration , 1999, SIAM J. Sci. Comput..

[7]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[8]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[9]  Michael Möller,et al.  A Convex Model for Matrix Factorization and Dimensionality Reduction on Physical Space and Its Application to Blind Hyperspectral Unmixing , 2010 .

[10]  E. G. Gol'shtein,et al.  Modified Lagrangians in Convex Programming and their Generalizations , 1979 .

[11]  Simon Setzer,et al.  Split Bregman Algorithm, Douglas-Rachford Splitting and Frame Shrinkage , 2009, SSVM.

[12]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[13]  Wotao Yin,et al.  Alternating direction augmented Lagrangian methods for semidefinite programming , 2010, Math. Program. Comput..

[14]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[15]  Deep Medhi,et al.  Generalized proximal point algorithm for convex optimization , 1996 .

[16]  W. Hager,et al.  Large Scale Optimization : State of the Art , 1993 .

[17]  Bingsheng He,et al.  Linearized Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming , 2011 .

[18]  Gene H. Golub,et al.  Numerical solution of saddle point problems , 2005, Acta Numerica.

[19]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[20]  Osman Güer.  On the convergence of the proximal point algorithm for convex minimization , 1991 .

[21]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[22]  Wotao Yin,et al.  Second-order Cone Programming Methods for Total Variation-Based Image Restoration , 2005, SIAM J. Sci. Comput..

[23]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[24]  I. Konnov.  Combined Relaxation Methods for Variational Inequalities , 2000 .

[25]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[27]  W. Han,et al.  On the finite element method for mixed variational inequalities arising in elastoplasticity , 1995 .

[28]  Xiaoming Yuan,et al.  Alternating algorithms for total variation image reconstruction from random projections , 2012 .

[29]  Yunhai Xiao,et al.  A Fast Algorithm for Total Variation Image Reconstruction from Random Projections , 2010 .

[30]  Michael K. Ng,et al.  Fast Image Restoration Methods for Impulse and Gaussian Noises Removal , 2009, IEEE Signal Processing Letters.

[31]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[32]  Bingsheng He,et al.  A customized proximal point algorithm for convex minimization with linear constraints , 2013, Comput. Optim. Appl..

[33]  Karl Meerbergen,et al.  The Quadratic Eigenvalue Problem , 2001, SIAM Rev..

[34]  Michel Fortin,et al.  Mixed and Hybrid Finite Element Methods , 2011, Springer Series in Computational Mathematics.

[35]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[36]  M. Korda.  Grundlagen und Verfahren , 1999 .

[37]  Raymond H. Chan,et al.  Alternating Direction Method for Image Inpainting in Wavelet Domains , 2011, SIAM J. Imaging Sci..

[38]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[39]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[40]  Masao Fukushima,et al.  Quasi-variational inequalities, generalized Nash equilibria, and multi-leader-follower games , 2009, Comput. Manag. Sci..

[41]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[42]  Pablo A. Parrilo,et al.  Rank-Sparsity Incoherence for Matrix Decomposition , 2009, SIAM J. Optim..

[43]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[44]  B. Lemaire.  Saddle-point problems in partial differential equations and applications to linear quadratic differential games , 1973 .

[45]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[46]  Daniel Cremers,et al.  An algorithm for minimizing the Mumford-Shah functional , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[47]  J. Burke,et al.  A Variable Metric Proximal Point Algorithm for Monotone Operators , 1999 .

[48]  Bingsheng He,et al.  Solving Large-Scale Least Squares Semidefinite Programming by Alternating Direction Methods , 2011, SIAM J. Matrix Anal. Appl..

[49]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[50]  Andrzej Ruszczynski,et al.  Parallel decomposition of multistage stochastic programming problems , 1993, Math. Program..

[51]  Xiaoming Yuan,et al.  A contraction method with implementable proximal regularization for linearly constrained convex programming , 2011 .

[52]  Junfeng Yang,et al.  Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization , 2012, Math. Comput..

[53]  Su Zhang,et al.  A modified alternating direction method for convex quadratically constrained quadratic semidefinite programs , 2010, Eur. J. Oper. Res..

[54]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[55]  Xiaoming Yuan,et al.  A proximal point algorithm revisit on the alternating direction method of multipliers , 2013 .

[56]  R. Rockafellar.  Convex Analysis: (pms-28) , 1970 .

[57]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[58]  Michael Möller,et al.  A Convex Model for Nonnegative Matrix Factorization and Dimensionality Reduction on Physical Space , 2011, IEEE Transactions on Image Processing.

[59]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[60]  Ernie Esser,et al.  Applications of Lagrangian-Based Alternating Direction Methods and Connections to Split Bregman , 2009 .

[61]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[62]  M. J. D. Powell,et al.  A method for nonlinear constraints in minimization problems , 1969 .

[63]  Laurent Condat.  A generic first-order primal-dual method for convex optimization involving Lipschitzian, proximable and linear composite terms , 2011 .

[64]  Bingsheng He,et al.  Some convergence properties of a method of multipliers for linearly constrained monotone variational inequalities , 1998, Oper. Res. Lett..

[65]  Xiaoming Yuan,et al.  A relaxed customized proximal point algorithm for separable convex programming , 2011 .

[66]  Xiaoming Yuan,et al.  LINEARIZED ALTERNATING DIRECTION METHOD FOR CONSTRAINED LINEAR LEAST-SQUARES PROBLEM , 2011 .

[67]  C. Ha.  A generalization of the proximal point algorithm , 1990 .

[68]  Stephen P. Boyd,et al.  Least-Squares Covariance Matrix Adjustment , 2005, SIAM J. Matrix Anal. Appl..

[69]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[70]  Bingsheng He,et al.  Proximal-Point Algorithm Using a Linear Proximal Term , 2009 .

[71]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[72]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[73]  B. Reddy.  Mixed variational inequalities arising in elastoplasticity , 1992 .

[74]  Jie Sun,et al.  An alternating direction method for solving convex nonlinear semidefinite programming problems , 2013 .

[75]  Muhammad Aslam Noor,et al.  Quasi variational inequalities , 1988 .

[76]  Michael K. Ng,et al.  Solving Constrained Total-variation Image Restoration and Reconstruction Problems via Alternating Direction Methods , 2010, SIAM J. Sci. Comput..

[77]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[78]  Xavier Bresson,et al.  Bregmanized Nonlocal Regularization for Deconvolution and Sparse Reconstruction , 2010, SIAM J. Imaging Sci..

[79]  Michael K. Ng,et al.  Inexact Alternating Direction Methods for Image Recovery , 2011, SIAM J. Sci. Comput..

[80]  Gilles Aubert,et al.  Efficient Schemes for Total Variation Minimization Under Constraints in Image Processing , 2009, SIAM J. Sci. Comput..

[81]  Igor V. Konnov,et al.  Partial proximal point method for nonmonotone equilibrium problems , 2006, Optim. Methods Softw..

[82]  Messaoud Bounkhel,et al.  Quasi-Variational Inequalities , 2012 .

[83]  Masao Fukushima,et al.  Application of the alternating direction method of multipliers to separable convex programming problems , 1992, Comput. Optim. Appl..

[84]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[85]  John Wright,et al.  RASL: Robust Alignment by Sparse and Low-Rank Decomposition for Linearly Correlated Images , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[86]  James V. Burke,et al.  On the superlinear convergence of the variable metric proximal point algorithm using Broyden and BFGS matrix secant updating , 2000, Math. Program..

[87]  J. Frédéric Bonnans,et al.  A family of variable metric proximal methods , 1995, Math. Program..

[88]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[89]  Robert R. Meyer,et al.  A variable-penalty alternating directions method for convex optimization , 1998, Math. Program..

[90]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[91]  Xiaoming Yuan,et al.  Matrix completion via an alternating direction method , 2012 .

[92]  Alexandre d'Aspremont,et al.  First-Order Methods for Sparse Covariance Selection , 2006, SIAM J. Matrix Anal. Appl..

[93]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[94]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[95]  Bingsheng He,et al.  Alternating directions based contraction method for generally separable linearly constrained convex programming problems , 2009 .

[96]  Xiaoming Yuan,et al.  Linearized Alternating Direction Method of Multipliers for Constrained Linear Least-Squares Problem , 2012 .

[97]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[98]  Raymond H. Chan,et al.  Journal of Computational and Applied Mathematics a Reduced Newton Method for Constrained Linear Least-squares Problems , 2022 .

[99]  Bingsheng He,et al.  On the O(1/t) convergence rate of the projection and contraction methods for variational inequalities with Lipschitz continuous monotone operators , 2013, Computational Optimization and Applications.

[100]  Masao Fukushima,et al.  Some Reformulations and Applications of the Alternating Direction Method of Multipliers , 1994 .

[101]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[102]  Renato D. C. Monteiro,et al.  Iteration-Complexity of Block-Decomposition Algorithms and the Alternating Direction Method of Multipliers , 2013, SIAM J. Optim..

[103]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[104]  Xiangfeng Wang,et al.  The Linearized Alternating Direction Method of Multipliers for Dantzig Selector , 2012, SIAM J. Sci. Comput..