A Primer on Monotone Operator Methods

This tutorial paper presents the basic notation and results of monotone operators and operator splitting methods, with a focus on convex optimization. A very wide variety of algorithms, ranging from classical to recently developed, can be derived in a uniform way. The approach is to pose the original problem to be solved as one of finding a zero of an appropriate monotone operator; this problem in turn is then posed as one of finding a fixed point of a related operator, which is done using the fixed point iteration. A few basic convergence results then tell us conditions under which the method converges, and, in some cases, how fast. This approach can be traced back to the 1960s and 1970s, and is still an active area of research. This primer is a self-contained gentle introduction to the topic.

[1]  É. Picard Mémoire sur la théorie des équations aux dérivées partielles et la méthode des approximations successives , 1890 .

[2]  S. Banach Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales , 1922 .

[3]  W. R. Mann,et al.  Mean value methods in iteration , 1953 .

[4]  H. H. Rachford,et al.  The Numerical Solution of Parabolic and Elliptic Differential Equations , 1955 .

[5]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[6]  E. E. Osborne On pre-conditioning matrices , 1959, ACM '59.

[7]  W. Cheney,et al.  Proximity maps for convex sets , 1959 .

[8]  A. A. Goldstein,et al.  Newton's method for convex programming and Tchebycheff approximation , 1959, Numerische Mathematik.

[9]  G. Minty Monotone networks , 1960, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[10]  J. E. Kelley,et al.  The Cutting-Plane Method for Solving Convex Programs , 1960 .

[11]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[12]  G. Minty On the maximal domain of a ``monotone'' function. , 1961 .

[13]  G. Minty Monotone (nonlinear) operators in Hilbert space , 1962 .

[14]  F. Browder,et al.  VARIATIONAL BOUNDARY VALUE PROBLEMS FOR QUASI-LINEAR ELLIPTIC EQUATIONS OF ARBITRARY ORDER. , 1963, Proceedings of the National Academy of Sciences of the United States of America.

[15]  F. Browder Nonlinear elliptic boundary value problems , 1963 .

[16]  Felix E. Browder,et al.  The solvability of non-linear functional equations , 1963 .

[17]  G. Minty On the monotonicity of the gradient of a convex function. , 1964 .

[18]  Xiaomei Yang Rounding Errors in Algebraic Processes , 1964, Nature.

[19]  A. Goldstein Convex programming in Hilbert space , 1964 .

[20]  D. J. Newman,et al.  Location of the Maximum on Unimodal Surfaces , 1965, JACM.

[21]  J. H. Wilkinson,et al.  Iterative refinement of the solution of a positive definite system of equations , 1966 .

[22]  J. H. Wilkinson,et al.  Note on the iterative refinement of least squares solution , 1966 .

[23]  R. Rockafellar Characterization of the subdifferentials of convex functions , 1966 .

[24]  Boris Polyak,et al.  Constrained minimization methods , 1966 .

[25]  G. Zames On the input-output stability of time-varying nonlinear feedback systems Part one: Conditions derived using concepts of loop gain, conicity, and positivity , 1966 .

[26]  Boris Polyak,et al.  The method of projections for finding the common point of convex sets , 1967 .

[27]  M. Powell A method for nonlinear constraints in minimization problems , 1969 .

[28]  J Descloux Analyse numérique I , 1969 .

[29]  M. Hestenes Multiplier and gradient methods , 1969 .

[30]  R. Kellogg A nonlinear alternating direction method , 1969 .

[31]  B. Martinet,et al.  R'egularisation d''in'equations variationnelles par approximations successives , 1970 .

[32]  R. Rockafellar,et al.  On the maximal monotonicity of subdifferential mappings. , 1970 .

[33]  R. Rockafellar On the maximality of sums of nonlinear monotone operators , 1970 .

[34]  R. Rockafellar The multiplier method of Hestenes and Powell applied to convex programming , 1973 .

[35]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[36]  R. Rockafellar Monotone Operators and the Proximal Point Algorithm , 1976 .

[37]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[38]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[39]  G. M. Korpelevich The extragradient method for finding saddle points and other problems , 1976 .

[40]  丸山 徹 Convex Analysisの二,三の進展について , 1977 .

[41]  H. Brezis,et al.  Produits infinis de resolvantes , 1978 .

[42]  R. Rockafellar MONOTONE OPERATORS AND AUGMENTED LAGRANGIAN METHODS IN NONLINEAR PROGRAMMING , 1978 .

[43]  Gregory B. Passty Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[44]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[45]  D. Kinderlehrer,et al.  An introduction to variational inequalities and their applications , 1980 .

[46]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[47]  J. Spingarn Partial inverse of a monotone operator , 1983 .

[48]  D. Gabay Applications of the method of multipliers to variational inequalities , 1983 .

[49]  Jonathan E. Spingarn,et al.  Applications of the method of partial inverses to convex programming: Decomposition , 1985, Math. Program..

[50]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[51]  R. Dykstra,et al.  A Method for Finding Projections onto the Intersection of Convex Sets in Hilbert Spaces , 1986 .

[52]  Paul Tseng,et al.  Relaxation methods for problems with strictly convex separable costs and linear constraints , 1987, Math. Program..

[53]  Jonathan Eckstein,et al.  The Lions-Mercier splitting algorithm and the alternating direction method are instances of the proximal point algorithm , 1988 .

[54]  Jonathan Eckstein Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[55]  R. Phelps Convex Functions, Monotone Operators and Differentiability , 1989 .

[56]  S. J. Chung NP-Completeness of the linear complementarity problem , 1989 .

[57]  P. Tseng Dual ascent methods for problems with strictly convex costs and linear constraints: a unified approach , 1990 .

[58]  J. Francu MONOTONE OPERATORS A SURVEY DIRECTED TO APPLICATIONS TO DIFFERENTIAL EQUATIONS , 1990 .

[59]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[60]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[61]  Heinz H. Bauschke,et al.  On the method of cyclic projections for convex sets in Hilbert space , 1994 .

[62]  Heinz H. Bauschke,et al.  Dykstra's Alternating Projection Algorithm for Two Sets , 1994 .

[63]  Robert J. Vanderbei,et al.  Symmetric Quasidefinite Matrices , 1995, SIAM J. Optim..

[64]  Joseph R. Shinnerl,et al.  On the Stability of Cholesky Factorization for Symmetric Quasidefinite Systems , 1996, SIAM J. Matrix Anal. Appl..

[65]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[66]  R. Showalter Monotone operators in Banach space and nonlinear partial differential equations , 1996 .

[67]  Michael C. Ferris,et al.  Engineering and Economic Applications of Complementarity Problems , 1997, SIAM Rev..

[68]  Stephen M. Robinson,et al.  Composition duality and maximal monotonicity , 1999, Math. Program..

[69]  Teemu Pennanen,et al.  Dualization of Generalized Equations of Maximal Monotone Type , 1999, SIAM J. Optim..

[70]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[71]  J. Borwein,et al.  Convex Analysis And Nonlinear Optimization , 2000 .

[72]  Arkadi Nemirovski,et al.  Lectures on modern convex optimization - analysis, algorithms, and engineering applications , 2001, MPS-SIAM series on optimization.

[73]  M. Teboulle,et al.  Asymptotic cones and functions in optimization and variational inequalities , 2002 .

[74]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[75]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[76]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[77]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[78]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[79]  A. Banerjee Convex Analysis and Optimization , 2006 .

[80]  Arkadi Nemirovski,et al.  EFFICIENT METHODS IN CONVEX PROGRAMMING , 2007 .

[81]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[82]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[83]  S. Kakade,et al.  On the duality of strong convexity and strong smoothness : Learning applications and matrix regularization , 2009 .

[84]  Richard W. Cottle,et al.  Linear Complementarity Problem , 2009, Encyclopedia of Optimization.

[85]  Yoram Singer,et al.  Efficient Online and Batch Learning Using Forward Backward Splitting , 2009, J. Mach. Learn. Res..

[86]  John E. Mitchell,et al.  Cutting Plane Methods and Subgradient Methods , 2009 .

[87]  M. Sofonea,et al.  Variational Inequalities with Applications: A Study of Antiplane Frictional Contact Problems , 2009 .

[88]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[89]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[90]  Ralph E. Gomory,et al.  Outline of an Algorithm for Integer Solutions to Linear Programs and An Algorithm for the Mixed Integer Problem , 2010, 50 Years of Integer Programming.

[91]  J. Borwein,et al.  Convex Functions: Constructions, Characterizations and Counterexamples , 2010 .

[92]  Panos M. Pardalos,et al.  Convex optimization theory , 2010, Optim. Methods Softw..

[93]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[94]  Patrick L. Combettes,et al.  A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality , 2010, SIAM J. Optim..

[95]  M. Raydan,et al.  Alternating Projection Methods , 2011 .

[96]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[97]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[98]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[99]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[100]  P. L. Combettes,et al.  Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators , 2011, Set-Valued and Variational Analysis.

[101]  Jonathan Eckstein Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[102]  Stephen P. Boyd,et al.  Operator Splitting for Conic Optimization via Homogeneous Self-Dual Embedding , 2013 .

[103]  Heinz H. Bauschke,et al.  Projection Methods: Swiss Army Knives for Solving Feasibility and Best Approximation Problems with Halfspaces , 2013, 1301.4506.

[104]  M. Noor,et al.  PARALLEL RESOLVENT METHODS FOR SYSTEM OF GENERAL VARIATIONAL INCLUSIONS , 2013 .

[105]  Ying Xiong Nonlinear Optimization , 2014 .

[106]  Stephen P. Boyd,et al.  Metric Selection in Douglas-Rachford Splitting and ADMM , 2014 .

[107]  Jonathan M. Borwein,et al.  Applications of convex analysis within mathematics , 2013, Math. Program..

[108]  P. L. Combettes,et al.  Compositions and convex combinations of averaged nonexpansive operators , 2014, 1407.5100.

[109]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[110]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[111]  Stephen P. Boyd,et al.  Block splitting for distributed optimization , 2013, Mathematical Programming Computation.

[112]  Pontus Giselsson,et al.  Tight global linear convergence rate bounds for Douglas–Rachford splitting , 2015, Journal of Fixed Point Theory and Applications.

[113]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[114]  Stephen P. Boyd,et al.  Metric selection in fast dual forward-backward splitting , 2015, Autom..

[115]  Dimitri P. Bertsekas,et al.  Convex Optimization Algorithms , 2015 .

[116]  Phillipp Kaestner,et al.  Linear And Nonlinear Programming , 2016 .

[117]  Joyati Debnath,et al.  Calculus Volume 2 , 2016 .

[118]  Ming Yan,et al.  Self Equivalence of the Alternating Direction Method of Multipliers , 2014, 1407.7400.

[119]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[120]  RockaJellm MONOTONE OPERATORS ASSOCIATED WITH SADDLE . FUNCTIONS AND MINIMAX PROBLEMS R . 1 ' , 2022 .