Stochastic Quasi-Fejér Block-Coordinate Fixed Point Iterations with Random Sweeping

This work proposes block-coordinate fixed point algorithms with applications to nonlinear analysis and optimization in Hilbert spaces. The asymptotic analysis relies on a notion of stochastic quasi-Fejer monotonicity, which is thoroughly investigated. The iterative methods under consideration feature random sweeping rules to select arbitrarily the blocks of variables that are activated over the course of the iterations and they allow for stochastic errors in the evaluation of the operators. Algorithms using quasi-nonexpansive operators or compositions of averaged nonexpansive operators are constructed, and weak and strong convergence results are established for the sequences they generate. As a by-product, novel block-coordinate operator splitting methods are obtained for solving structured monotone inclusion and convex minimization problems. In particular, the proposed framework leads to random block-coordinate versions of the Douglas--Rachford and forward-backward algorithms and of some of their variant...

[1]  B. Pettis.  On integration in vector spaces , 1938 .

[2]  J. Moreau.  Fonctions convexes duales et points proximaux dans un espace hilbertien , 1962 .

[3]  W. Petryshyn.  Construction of fixed points of demicompact mappings in Hilbert space , 1966 .

[4]  Yu. M. Ermol’ev.  On the method of generalized stochastic gradients and quasi-Féjer sequences , 1969 .

[5]  M. Sibony.  Méthodes itératives pour les équations et inéquations aux dérivées partielles non linéaires de type monotone , 1970 .

[6]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[7]  J. Cea.  Optimisation : théorie et algorithmes , 1971 .

[8]  A. Auslender.  Méthodes numériques pour la décomposition et la minimisation de fonctions non différentiables , 1971 .

[9]  Yu. M. Ermol’ev.  On convergence of quasi-Féjer sequences , 1971 .

[10]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[11]  B. Mercier.  Topics in Finite Element Solution of Elliptic Problems , 1979 .

[12]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[13]  M. Loève,et al.  Probability Theory II (4th ed.). , 1979 .

[14]  Felipe Acker,et al.  Convergence d'un schéma de minimisation alternée , 1980 .

[15]  M. Fortin,et al.  Augmented Lagrangian methods : applications to the numerical solution of boundary-value problems , 1983 .

[16]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[17]  H. Robbins,et al.  A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications , 1985 .

[18]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[19]  Paul Tseng,et al.  Further applications of a splitting algorithm to decomposition in variational inequalities and convex programming , 1990, Math. Program..

[20]  P. Tseng.  Applications of splitting algorithm to decomposition in convex programming and variational inequalities , 1991 .

[21]  M. Talagrand,et al.  Probability in Banach Spaces: Isoperimetry and Processes , 1991 .

[22]  Nonlinear functional analysis and its applications, part I: Fixed-point theorems , 1991 .

[23]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[24]  R. D. Murphy,et al.  Iterative solution of nonlinear equations , 1994 .

[25]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[26]  P. L. Combettes,et al.  Quasi-Fejérian Analysis of Some Optimization Algorithms , 2001 .

[27]  Heinz H. Bauschke,et al.  A Weak-to-Strong Convergence Principle for Fejé-Monotone Methods in Hilbert Spaces , 2001, Math. Oper. Res..

[28]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[29]  Hein Hundal.  An alternating projection that does not converge in norm , 2004 .

[30]  Heinz H. Bauschke,et al.  The asymptotic behavior of the composition of two resolvents , 2005, Nonlinear Analysis: Theory, Methods & Applications.

[31]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[32]  Markku Kallio,et al.  A splitting method for stochastic programs , 2006, Ann. Oper. Res..

[33]  Kengy Barty,et al.  Hilbert-Valued Perturbed Subgradient Algorithms , 2007, Math. Oper. Res..

[34]  I. I. Eremin,et al.  Fejér processes in theory and practice: Recent results , 2009 .

[35]  P. L. Combettes,et al.  Iterative construction of the resolvent of a sum of maximal monotone operators , 2009 .

[36]  Patrick L. Combettes,et al.  Fejér Monotonicity in Convex Optimization , 2009, Encyclopedia of Optimization.

[37]  Luis,et al.  Convex Variational Formulation with Smooth Coupling for Multicomponent Signal Decomposition and Recovery , 2009 .

[38]  Heinz H. Bauschke,et al.  Characterizing arbitrarily slow convergence in the method of alternating projections , 2007, Int. Trans. Oper. Res..

[39]  P. L. Combettes,et al.  Dualization of Signal Recovery Problems , 2009, 0907.0436.

[40]  Patrick L. Combettes,et al.  A Parallel Splitting Method for Coupled Monotone Inclusions , 2009, SIAM J. Control. Optim..

[41]  Peter Orbanz.  Probability Theory II , 2011 .

[42]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[43]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[44]  P. L. Combettes,et al.  There is no variational characterization of the cycles in the method of periodic projections , 2011, 1102.1378.

[45]  Ernesto De Vito,et al.  A consistent algorithm to solve Lasso, elastic-net and Tikhonov regularization , 2011, J. Complex..

[46]  Luis M Briceño-Arias.  Problèmes d'inclusions couplées : Éclatement, algorithmes et applications , 2011 .

[47]  Heinz H. Bauschke,et al.  Firmly Nonexpansive Mappings and Maximally Monotone Operators: Correspondence and Duality , 2011, 1101.4688.

[48]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[49]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[50]  L. Briceño-Arias.  Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions , 2012, 1212.5942.

[51]  A. Cegielski.  Iterative Methods for Fixed Point Problems in Hilbert Spaces , 2012 .

[52]  J. Pesquet,et al.  A Parallel Inertial Proximal Optimization Method , 2012 .

[53]  L. Briceño-Arias.  A Douglas–Rachford splitting method for solving equilibrium problems , 2011, 1110.1670.

[54]  P. L. Combettes,et al.  Monotone Operator Methods for Nash Equilibria in Non-potential Games , 2011, 1106.0144.

[55]  Pascal Bianchi,et al.  Asynchronous distributed optimization using a randomized alternating direction method of multipliers , 2013, 52nd IEEE Conference on Decision and Control.

[56]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[57]  Luca Baldassarre,et al.  Accelerated and Inexact Forward-Backward Algorithms , 2013, SIAM J. Optim..

[58]  Radu Ioan Bot,et al.  A Douglas-Rachford Type Primal-Dual Method for Solving Inclusions with Mixtures of Composite and Parallel-Sum Type Monotone Operators , 2012, SIAM J. Optim..

[59]  Stephen J. Wright,et al.  Optimization for Machine Learning , 2013 .

[60]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[61]  Patrick L. Combettes,et al.  Systems of Structured Monotone Inclusions: Duality, Algorithms, and Applications , 2012, SIAM J. Optim..

[62]  Mohamed-Jalal Fadili,et al.  A Generalized Forward-Backward Splitting , 2011, SIAM J. Imaging Sci..

[63]  C. Byrne.  Iterative Optimization in Inverse Problems , 2014 .

[64]  Patrick L. Combettes,et al.  A primal-dual method of partial inverses for composite inclusions , 2013, Optim. Lett..

[65]  Ion Necoara,et al.  A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints , 2013, Comput. Optim. Appl..

[66]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[67]  Gabriel Peyré,et al.  Optimal Transport with Proximal Splitting , 2013, SIAM J. Imaging Sci..

[68]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[69]  L. Rosasco,et al.  A Stochastic forward-backward splitting method for solving monotone inclusions in Hilbert spaces , 2014, 1403.7999.

[70]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[71]  Lin Xiao,et al.  On the complexity analysis of randomized block-coordinate descent methods , 2013, Mathematical Programming.

[72]  Heinz H. Bauschke,et al.  Optimal Rates of Linear Convergence of Relaxed Alternating Projections and Generalized Douglas-Rachford Methods for Two Subspaces , 2015, Numerical Algorithms.

[73]  Radu Ioan Bot,et al.  On the convergence rate improvement of a primal-dual splitting algorithm for solving monotone inclusion problems , 2013, Mathematical Programming.

[74]  R. Boţ,et al.  Convergence rates for forward-backward dynamical systems associated with strongly monotone inclusions , 2015, 1504.01863.

[75]  P. L. Combettes,et al.  Stochastic Approximations and Perturbations in Forward-Backward Splitting for Monotone Operators , 2015, 1507.07095.

[76]  Heinz H. Bauschke,et al.  Proximal point algorithm, Douglas-Rachford algorithm and alternating projections: a case study , 2015, 1501.06603.

[77]  Peter Richtárik,et al.  On optimal probabilities in stochastic coordinate descent methods , 2013, Optim. Lett..

[78]  Émilie Chouzenoux,et al.  A block coordinate variable metric forward–backward algorithm , 2016, Journal of Global Optimization.

[79]  Gersende Fort,et al.  On Perturbed Proximal Gradient Algorithms , 2014, J. Mach. Learn. Res..