Stochastic quasi-Fejér block-coordinate fixed point iterations with random sweeping II: mean-square and linear convergence

Combettes and Pesquet (SIAM J Optim 25:1221–1248, 2015) investigated the almost sure weak convergence of block-coordinate fixed point algorithms and discussed their applications to nonlinear analysis and optimization. This algorithmic framework features random sweeping rules to select arbitrarily the blocks of variables that are activated over the course of the iterations and it allows for stochastic errors in the evaluation of the operators. The present paper establishes results on the mean-square and linear convergence of the iterates. Applications to monotone operator splitting and proximal optimization algorithms are presented.

[1]  Émilie Chouzenoux,et al.  A block coordinate variable metric forward–backward algorithm , 2016, Journal of Global Optimization.

[2]  P. L. Combettes,et al.  Stochastic Approximations and Perturbations in Forward-Backward Splitting for Monotone Operators , 2015, 1507.07095.

[3]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[4]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[5]  Patrick L. Combettes,et al.  Systems of Structured Monotone Inclusions: Duality, Algorithms, and Applications , 2012, SIAM J. Optim..

[6]  Peter Richtárik,et al.  On optimal probabilities in stochastic coordinate descent methods , 2013, Optim. Lett..

[7]  C. Byrne.  Iterative Optimization in Inverse Problems , 2014 .

[8]  M. Sibony.  Méthodes itératives pour les équations et inéquations aux dérivées partielles non linéaires de type monotone , 1970 .

[9]  B. Mercier.  Topics in Finite Element Solution of Elliptic Problems , 1979 .

[10]  Heinz H. Bauschke,et al.  A Weak-to-Strong Convergence Principle for Fejé-Monotone Methods in Hilbert Spaces , 2001, Math. Oper. Res..

[11]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[12]  Heinz H. Bauschke,et al.  Optimal Rates of Linear Convergence of Relaxed Alternating Projections and Generalized Douglas-Rachford Methods for Two Subspaces , 2015, Numerical Algorithms.

[13]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[14]  H. Robbins,et al.  A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications , 1985 .

[15]  D. Klatte.  Book review: Implicit Functions and Solution Mappings:A View from Variational Analysis. Second Edition. By A. L. Dontchev and R. T. Rockafellar. Springer, New York, 2014 , 2015 .

[16]  Kengy Barty,et al.  Hilbert-Valued Perturbed Subgradient Algorithms , 2007, Math. Oper. Res..

[17]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[18]  Paul Tseng,et al.  Further applications of a splitting algorithm to decomposition in variational inequalities and convex programming , 1990, Math. Program..

[19]  Nonlinear functional analysis and its applications, part I: Fixed-point theorems , 1991 .

[20]  J. Pesquet,et al.  A Parallel Inertial Proximal Optimization Method , 2012 .

[21]  Ernesto De Vito,et al.  A consistent algorithm to solve Lasso, elastic-net and Tikhonov regularization , 2011, J. Complex..

[22]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[23]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[24]  Ion Necoara,et al.  A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints , 2013, Comput. Optim. Appl..

[25]  A. Cegielski.  Iterative Methods for Fixed Point Problems in Hilbert Spaces , 2012 .

[26]  J. Moreau.  Fonctions convexes duales et points proximaux dans un espace hilbertien , 1962 .

[27]  P. L. Combettes,et al.  Quasi-Fejérian Analysis of Some Optimization Algorithms , 2001 .

[28]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[29]  Pascal Bianchi,et al.  Asynchronous distributed optimization using a randomized alternating direction method of multipliers , 2013, 52nd IEEE Conference on Decision and Control.

[30]  Luis,et al.  Convex Variational Formulation with Smooth Coupling for Multicomponent Signal Decomposition and Recovery , 2009 .

[31]  P. L. Combettes,et al.  There is no variational characterization of the cycles in the method of periodic projections , 2011, 1102.1378.

[32]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[33]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[34]  Luis M Briceño-Arias.  Problèmes d'inclusions couplées : Éclatement, algorithmes et applications , 2011 .

[35]  M. Loève,et al.  Probability Theory II (4th ed.). , 1979 .

[36]  L. Rosasco,et al.  A Stochastic forward-backward splitting method for solving monotone inclusions in Hilbert spaces , 2014, 1403.7999.

[37]  Patrick L. Combettes,et al.  Stochastic Quasi-Fejér Block-Coordinate Fixed Point Iterations with Random Sweeping , 2014 .

[38]  Patrick L. Combettes,et al.  A primal-dual method of partial inverses for composite inclusions , 2013, Optim. Lett..

[39]  Radu Ioan Bot,et al.  A Douglas-Rachford Type Primal-Dual Method for Solving Inclusions with Mixtures of Composite and Parallel-Sum Type Monotone Operators , 2012, SIAM J. Optim..

[40]  Yu. M. Ermol’ev.  On the method of generalized stochastic gradients and quasi-Féjer sequences , 1969 .

[41]  Yi Li,et al.  A Robust-Equitable Measure for Feature Ranking and Selection , 2017, J. Mach. Learn. Res..

[42]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[43]  A. Auslender.  Méthodes numériques pour la décomposition et la minimisation de fonctions non différentiables , 1971 .

[44]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[45]  Felipe Acker,et al.  Convergence d'un schéma de minimisation alternée , 1980 .

[46]  B. Pettis.  On integration in vector spaces , 1938 .

[47]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[48]  Peter Orbanz.  Probability Theory II , 2011 .

[49]  Markku Kallio,et al.  A splitting method for stochastic programs , 2006, Ann. Oper. Res..

[50]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[51]  M. Talagrand,et al.  Probability in Banach Spaces: Isoperimetry and Processes , 1991 .

[52]  Gersende Fort,et al.  On Perturbed Proximal Gradient Algorithms , 2014, J. Mach. Learn. Res..

[53]  J. Cea.  Optimisation : théorie et algorithmes , 1971 .

[54]  Jean-Yves Audibert.  Optimization for Machine Learning , 1995 .

[55]  Heinz H. Bauschke,et al.  The asymptotic behavior of the composition of two resolvents , 2005, Nonlinear Analysis: Theory, Methods & Applications.

[56]  Gabriel Peyré,et al.  Optimal Transport with Proximal Splitting , 2013, SIAM J. Imaging Sci..

[57]  P. L. Combettes.  Iterative construction of the resolvent of a sum of maximal monotone operators , 2009 .

[58]  Patrick L. Combettes,et al.  A Parallel Splitting Method for Coupled Monotone Inclusions , 2009, SIAM J. Control. Optim..

[59]  P. Tseng.  Applications of splitting algorithm to decomposition in convex programming and variational inequalities , 1991 .

[60]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[61]  P. L. Combettes,et al.  Monotone Operator Methods for Nash Equilibria in Non-potential Games , 2011, 1106.0144.

[62]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[63]  I. I. Eremin,et al.  Fejér processes in theory and practice: Recent results , 2009 .

[64]  Heinz H. Bauschke,et al.  Firmly Nonexpansive Mappings and Maximally Monotone Operators: Correspondence and Duality , 2011, 1101.4688.

[65]  R. Boţ,et al.  Convergence rates for forward-backward dynamical systems associated with strongly monotone inclusions , 2015, 1504.01863.

[66]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[67]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[68]  Luca Baldassarre,et al.  Accelerated and Inexact Forward-Backward Algorithms , 2013, SIAM J. Optim..

[69]  Radu Ioan Bot,et al.  On the convergence rate improvement of a primal-dual splitting algorithm for solving monotone inclusion problems , 2013, Mathematical Programming.

[70]  Mohamed-Jalal Fadili,et al.  A Generalized Forward-Backward Splitting , 2011, SIAM J. Imaging Sci..

[71]  Lorenzo Rosasco,et al.  Stochastic Forward–Backward Splitting for Monotone Inclusions , 2016, J. Optim. Theory Appl..