Stochastic Forward Douglas-Rachford Splitting for Monotone Inclusions

We propose a stochastic Forward-Douglas-Rachford Splitting framework for finding a zero point of the sum of three maximally monotone operators in real separable Hilbert space, where one of the operators is cocoercive. We characterize the rate of convergence in expectation in the case of strongly monotone operators. We provide guidance on step-size sequences that achieve this rate, even if the strong convexity parameter is unknown.

[1]  B. Mercier Lectures on Topics in Finite Element Solution of Elliptic Problems , 1980 .

[2]  B. Mercier Topics in Finite Element Solution of Elliptic Problems , 1979 .

[3]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[4]  M. Talagrand,et al.  Probability in Banach Spaces: Isoperimetry and Processes , 1991 .

[5]  Allan Borodin,et al.  Can We Learn to Beat the Best Stock , 2003, NIPS.

[6]  O. SIAMJ.,et al.  PROX-METHOD WITH RATE OF CONVERGENCE O(1/t) FOR VARIATIONAL INEQUALITIES WITH LIPSCHITZ CONTINUOUS MONOTONE OPERATORS AND SMOOTH CONVEX-CONCAVE SADDLE POINT PROBLEMS∗ , 2004 .

[7]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[8]  Jean-Marie Monnez,et al.  Almost sure convergence of stochastic gradient processes with matrix step sizes , 2006 .

[9]  Jean-Marie Monnez,et al.  Almost sure convergence of a stochastic approximation process in a convex set , 2007 .

[10]  Kengy Barty,et al.  Hilbert-Valued Perturbed Subgradient Algorithms , 2007, Math. Oper. Res..

[11]  J.-C. Pesquet,et al.  A Douglas–Rachford Splitting Approach to Nonsmooth Convex Variational Signal Recovery , 2007, IEEE Journal of Selected Topics in Signal Processing.

[12]  I. Daubechies,et al.  Sparse and stable Markowitz portfolios , 2007, Proceedings of the National Academy of Sciences.

[13]  Yoram Singer,et al.  Efficient Online and Batch Learning Using Forward Backward Splitting , 2009, J. Mach. Learn. Res..

[14]  James T. Kwok,et al.  Accelerated Gradient Methods for Stochastic Optimization and Online Learning , 2009, NIPS.

[15]  Patrick L. Combettes,et al.  A Parallel Splitting Method for Coupled Monotone Inclusions , 2009, SIAM J. Control. Optim..

[16]  Patrick L. Combettes,et al.  A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality , 2010, SIAM J. Optim..

[17]  Elad Hazan,et al.  An optimal algorithm for stochastic strongly-convex optimization , 2010, 1006.2425.

[18]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[19]  Eric Moulines,et al.  Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning , 2011, NIPS.

[20]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[21]  P. L. Combettes,et al.  Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators , 2011, Set-Valued and Variational Analysis.

[22]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[23]  L. Briceño-Arias Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions , 2012, 1212.5942.

[24]  Guanghui Lan,et al.  An optimal method for stochastic composite optimization , 2011, Mathematical Programming.

[25]  Saeed Ghadimi,et al.  Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework , 2012, SIAM J. Optim..

[26]  Euhanna Ghadimi,et al.  On the Optimal Step-size Selection for the Alternating Direction Method of Multipliers* , 2012 .

[27]  Ohad Shamir,et al.  Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization , 2011, ICML.

[28]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[29]  Mohamed-Jalal Fadili,et al.  A Generalized Forward-Backward Splitting , 2011, SIAM J. Imaging Sci..

[30]  É. Moulines,et al.  On stochastic proximal gradient algorithms , 2014 .

[31]  Pascal Bianchi,et al.  A stochastic coordinate descent primal-dual algorithm and applications , 2014, 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP).

[32]  Patrick L. Combettes,et al.  A forward-backward view of some primal-dual optimization methods in image recovery , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[33]  J. Pesquet,et al.  A Class of Randomized Primal-Dual Algorithms for Distributed Optimization , 2014, 1406.6404.

[34]  L. Rosasco,et al.  A Stochastic forward-backward splitting method for solving monotone inclusions in Hilbert spaces , 2014, 1403.7999.

[35]  Bang Công Vu,et al.  A Splitting Algorithm for Coupled System of Primal–Dual Monotone Inclusions , 2014, Journal of Optimization Theory and Applications.

[36]  Patrick L. Combettes,et al.  Stochastic Quasi-Fejér Block-Coordinate Fixed Point Iterations with Random Sweeping , 2014 .

[37]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[38]  Kristian Bredies,et al.  Preconditioned Douglas-Rachford Splitting Methods for Convex-concave Saddle-point Problems , 2015, SIAM J. Numer. Anal..

[39]  P. L. Combettes,et al.  Stochastic Approximations and Perturbations in Forward-Backward Splitting for Monotone Operators , 2015, 1507.07095.

[40]  L. Rosasco,et al.  A stochastic inertial forward–backward splitting algorithm for multivariate monotone inclusions , 2015, 1507.00848.

[41]  Lorenzo Rosasco,et al.  Stochastic Forward–Backward Splitting for Monotone Inclusions , 2016, J. Optim. Theory Appl..

[42]  Volkan Cevher,et al.  Stochastic Three-Composite Convex Minimization , 2017, NIPS.

[43]  Bang Công Vu,et al.  Almost sure convergence of the forward–backward–forward splitting algorithm , 2015, Optim. Lett..

[44]  Pascal Bianchi,et al.  A Coordinate Descent Primal-Dual Algorithm and Application to Distributed Asynchronous Optimization , 2014, IEEE Transactions on Automatic Control.

[45]  Yi Li,et al.  A Robust-Equitable Measure for Feature Ranking and Selection , 2017, J. Mach. Learn. Res..

[46]  Gersende Fort,et al.  On Perturbed Proximal Gradient Algorithms , 2014, J. Mach. Learn. Res..

[47]  Panagiotis Patrinos,et al.  Asymmetric forward–backward–adjoint splitting for solving monotone inclusions involving three operators , 2016, Comput. Optim. Appl..

[48]  L. Rosasco,et al.  Convergence of Stochastic Proximal Gradient Algorithm , 2014, Applied Mathematics & Optimization.

[49]  Ernest K. Ryu,et al.  Proximal-Proximal-Gradient Method , 2017, Journal of Computational Mathematics.