Douglas–Rachford splitting and ADMM for pathological convex optimization

Despite the vast literature on DRS and ADMM, there had been very little work analyzing their behavior under pathologies. Most analyses assume a primal solution exists, a dual solution exists, and strong duality holds. When these assumptions are not met, i.e., under pathologies, the theory often breaks down and the empirical performance may degrade significantly. In this paper, we establish that DRS only requires strong duality to work, in the sense that asymptotically iterates are approximately feasible and approximately optimal. We then translate the pathological analyses for DRS to pathological analyses for ADMM.

[1]  H. Attouch.  A General Duality Principle for the Sum of Two Operators 1 , 1996 .

[2]  Masakazu Muramatsu,et al.  A facial reduction algorithm for finding sparse SOS representations , 2010, Oper. Res. Lett..

[3]  Stefano Di Cairano,et al.  Infeasibility detection in alternating direction method of multipliers for convex quadratic programs , 2014, 53rd IEEE Conference on Decision and Control.

[4]  J. Borwein,et al.  Facial reduction for a cone-convex programming problem , 1981, Journal of the Australian Mathematical Society. Series A. Pure Mathematics and Statistics.

[5]  Dmitriy Drusvyatskiy,et al.  The Many Faces of Degeneracy in Conic Optimization , 2017, Found. Trends Optim..

[6]  Henry Wolkowicz,et al.  Strong Duality for Semidefinite Programming , 1997, SIAM J. Optim..

[7]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[8]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[9]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[10]  Heinz H. Bauschke,et al.  On the Range of the Douglas-Rachford Operator , 2014, Math. Oper. Res..

[11]  Henry Wolkowicz,et al.  Strong duality and minimal representations for cone optimization , 2012, Computational Optimization and Applications.

[12]  Heinz H. Bauschke,et al.  The Douglas-Rachford Algorithm for Two (Not Necessarily Intersecting) Affine Subspaces , 2015, SIAM J. Optim..

[13]  Shuzhong Zhang,et al.  Duality Results for Conic Convex Programming , 1997 .

[14]  Heinz H. Bauschke,et al.  Finding best approximation pairs relative to two closed convex sets in Hilbert spaces , 2004, J. Approx. Theory.

[15]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[16]  Gábor Pataki.  A Simple Derivation of a Facial Reduction Algorithm and Extended Dual Systems , .

[17]  Z. Luo,et al.  Conic convex programming and self-dual embedding , 1998 .

[18]  Wotao Yin,et al.  An Envelope for Davis–Yin Splitting and Strict Saddle-Point Avoidance , 2018, J. Optim. Theory Appl..

[19]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[20]  Jonathan M. Borwein,et al.  The Cyclic Douglas-Rachford Method for Inconsistent Feasibility Problems , 2013, 1310.2195.

[21]  Heinz H. Bauschke,et al.  The method of cyclic projections for closed convex sets in Hilbert space , 1997 .

[22]  Bolor Jargalsaikhan,et al.  Linear conic programming: genericity and stability , 2015 .

[23]  Masakazu Muramatsu,et al.  Strange behaviors of interior-point methods for solving semidefinite programming problems in polynomial optimization , 2012, Comput. Optim. Appl..

[24]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[25]  Damek Davis,et al.  Convergence Rate Analysis of Primal-Dual Splitting Schemes , 2014, SIAM J. Optim..

[26]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[27]  A. Tucker,et al.  Linear Inequalities And Related Systems , 1956 .

[28]  A. Pazy.  Asymptotic behavior of contractions in hilbert space , 1971 .

[29]  Hayato Waki,et al.  How to generate weakly infeasible semidefinite programs via Lasserre’s relaxations for polynomial optimization , 2011, Optimization Letters.

[30]  Kim-Chuan Toh,et al.  On the equivalence of inexact proximal ALM and ADMM for a class of convex composite programming , 2018, Math. Program..

[31]  Ernest K. Ryu.  Cosmic divergence, weak cosmic convergence, and fixed points at infinity , 2017, Journal of Fixed Point Theory and Applications.

[32]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[33]  Dmitriy Drusvyatskiy,et al.  A note on alternating projections for ill-posed semidefinite feasibility problems , 2017, Math. Program..

[34]  Heinz H. Bauschke,et al.  Affine Nonexpansive Operators, Attouch–Théra Duality and the Douglas–Rachford Algorithm , 2016, 1603.09418.

[35]  H. H. Rachford,et al.  The Numerical Solution of Parabolic and Elliptic Differential Equations , 1955 .

[36]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[37]  Minghui Liu,et al.  Exact duals and short certificates of infeasibility and weak infeasibility in conic linear programming , 2015, Math. Program..

[38]  Wotao Yin,et al.  A new use of Douglas–Rachford splitting for identifying infeasible, unbounded, and pathological conic programs , 2019, Math. Program..

[39]  M. Fortin,et al.  On decomposition - coordination methods using an augmented Lagrangian , 1983 .

[40]  Michael J. Todd,et al.  Infeasible-start primal-dual methods and infeasibility detectors for nonlinear programming problems , 1999, Math. Program..

[41]  Heinz H. Bauschke,et al.  On the Douglas–Rachford algorithm , 2016, Mathematical Programming.

[42]  J. Borwein,et al.  Regularizing the Abstract Convex Program , 1981 .

[43]  Maretsugu Yamasaki.  Some generalizations of duality theorems in mathematical programming problems , 1969 .

[44]  Wotao Yin,et al.  A New Use of Douglas-Rachford Splitting and ADMM for Identifying Infeasible, Unbounded, and Pathological Conic Programs , 2017, ArXiv.

[45]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[46]  Pablo A. Parrilo,et al.  Computation with Polynomial Equations and Inequalities Arising in Combinatorial Optimization , 2009, 0909.0808.

[47]  Gábor Pataki,et al.  Sieve-SDP: a simple facial reduction algorithm to preprocess semidefinite programs , 2017, Mathematical Programming Computation.

[48]  Heinz H. Bauschke,et al.  On a result of Pazy concerning the asymptotic behaviour of nonexpansive mappings , 2015, 1505.04129.

[49]  Motakuri V. Ramana,et al.  An exact duality theory for semidefinite programming and its complexity implications , 1997, Math. Program..

[50]  Heinz H. Bauschke,et al.  Generalized Solutions for the Sum of Two Maximally Monotone Operators , 2013, SIAM J. Control. Optim..

[51]  M. Théra,et al.  Generalized sums of monotone operators , 1999 .

[52]  Simon P. Schurr,et al.  Preprocessing and Regularization for Degenerate Semidefinite Programs , 2013 .

[53]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[54]  J. Borwein,et al.  Characterizations of optimality without constraint qualification for the abstract convex program , 1982 .

[55]  R. Rockafellar.  Convex Analysis: (pms-28) , 1970 .

[56]  Matthew K. Tam,et al.  DOUGLAS–RACHFORD FEASIBILITY METHODS FOR MATRIX COMPLETION PROBLEMS , 2013, The ANZIAM Journal.

[57]  Frank Permenter,et al.  Solving Conic Optimization Problems via Self-Dual Embedding and Facial Reduction: A Unified Approach , 2017, SIAM J. Optim..

[58]  Pablo A. Parrilo,et al.  Basis selection for SOS programs via facial reduction and polyhedral approximations , 2014, 53rd IEEE Conference on Decision and Control.

[59]  Panagiotis Patrinos,et al.  Forward–backward quasi-Newton methods for nonsmooth optimization problems , 2016, Computational Optimization and Applications.

[60]  Paul Tseng.  Some convex programs without a duality gap , 2009, Math. Program..

[61]  R. Rockafellar.  Conjugate Duality and Optimization , 1987 .

[62]  Johan Löfberg,et al.  Pre- and Post-Processing Sum-of-Squares Programs in Practice , 2009, IEEE Transactions on Automatic Control.

[63]  J. Farkas.  Theorie der einfachen Ungleichungen. , 1902 .

[64]  Alberto Bemporad,et al.  Douglas-rachford splitting: Complexity estimates and accelerated variants , 2014, 53rd IEEE Conference on Decision and Control.

[65]  G. Pataki.  Strong Duality in Conic Linear Programming: Facial Reduction and Extended Duals , 2013, 1301.7717.

[66]  Gábor Pataki,et al.  Bad Semidefinite Programs: They All Look the Same , 2011, SIAM J. Optim..

[67]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[68]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[69]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[70]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[71]  Bruno F. Lourenço,et al.  Solving SDP completely with an interior point oracle , 2021, Optim. Methods Softw..

[72]  Jonathan Eckstein,et al.  Understanding the Convergence of the Alternating Direction Method of Multipliers: Theoretical and Computational Perspectives , 2015 .

[73]  M. Fortin,et al.  Augmented Lagrangian methods : applications to the numerical solution of boundary-value problems , 1983 .

[74]  R. Kellogg.  A nonlinear alternating direction method , 1969 .

[75]  B. Mercier.  Inequations variationnelles de la mécanique , 1980 .

[76]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[77]  Walaa M. Moursi,et al.  The Forward–Backward Algorithm and the Normal Problem , 2016, J. Optim. Theory Appl..

[78]  K. S. Kretschmer,et al.  Programmes in Paired Spaces , 1961, Canadian Journal of Mathematics.

[79]  Heinz H. Bauschke,et al.  Attouch-Théra duality revisited: Paramonotonicity and operator splitting , 2011, J. Approx. Theory.

[80]  Masakazu Muramatsu,et al.  A structural geometrical analysis of weakly infeasible SDPs , 2015 .

[81]  Patrick L. Combettes,et al.  Monotone operator theory in convex optimization , 2018, Math. Program..

[82]  Ming Yan,et al.  Self Equivalence of the Alternating Direction Method of Multipliers , 2014, 1407.7400.

[83]  Dimitri P. Bertsekas,et al.  Convex Optimization Theory , 2009 .

[84]  A. Bemporad,et al.  Forward-backward truncated Newton methods for convex composite optimization , 2014, 1402.6655.

[85]  Pablo A. Parrilo,et al.  Partial facial reduction: simplified, equivalent SDPs via approximations of the PSD cone , 2014, Math. Program..

[86]  W. Fenchel.  Convex cones, sets, and functions , 1953 .

[87]  Stephen P. Boyd,et al.  Infeasibility Detection in the Alternating Direction Method of Multipliers for Convex Optimization , 2018, Journal of Optimization Theory and Applications.

[88]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[89]  Michel Théra,et al.  Enlargements and Sums of Monotone Operators , 2001 .

[90]  Masakazu Muramatsu,et al.  Facial Reduction Algorithms for Conic Optimization Problems , 2012, Journal of Optimization Theory and Applications.

[91]  Jonathan M. Borwein,et al.  Global behavior of the Douglas–Rachford method for a nonconvex feasibility problem , 2015, J. Glob. Optim..

[92]  Paul Tseng,et al.  Hankel Matrix Rank Minimization with Applications to System Identification and Realization , 2013, SIAM J. Matrix Anal. Appl..

[93]  Kim-Chuan Toh,et al.  A note on the convergence of ADMM for linearly constrained convex optimization problems , 2015, Computational Optimization and Applications.

[94]  H. Attouch,et al.  Variational Sum of Monotone Operators , 1994 .

[95]  James Renegar,et al.  Computing approximate solutions for convex conic systems of constraints , 2000, Math. Program..

[96]  Stephen P. Boyd,et al.  OSQP: an operator splitting solver for quadratic programs , 2017, 2018 UKACC 12th International Conference on Control (CONTROL).

[97]  Mattias Fält,et al.  Envelope Functions: Unifications and Further Properties , 2016, J. Optim. Theory Appl..