The method of Bregman projections in deterministic and stochastic convex feasibility problems

In this work we study the method of Bregman projections for deterministic and stochastic convex feasibility problems with three types of control sequences for the selection of sets during the algorithmic procedure: greedy, random, and adaptive random. We analyze in depth the case of affine feasibility problems showing that the iterates generated by the proposed methods converge Q-linearly and providing also explicit global and local rates of convergence. This work generalizes from one hand recent developments in randomized methods for the solution of linear systems based on orthogonal projection methods. On the other hand, our results yield global and local Q-linear rates of convergence for the Sinkhorn and Greenhorn algorithms in discrete entropic-regularized optimal transport, for the first time, even in the multimarginal setting.

[1]  Heinz H. Bauschke,et al.  Bregman Monotone Optimization Algorithms , 2003, SIAM J. Control. Optim..

[2]  Inderjit S. Dhillon,et al.  Matrix Nearness Problems with Bregman Divergences , 2007, SIAM J. Matrix Anal. Appl..

[3]  Heinz H. Bauschke,et al.  Iterating Bregman Retractions , 2002, SIAM J. Optim..

[4]  L. Bregman.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming , 1967 .

[5]  R. Durrett.  Probability: Measure Theory , 2010 .

[6]  Peter Richtárik,et al.  Randomized Iterative Methods for Linear Systems , 2015, SIAM J. Matrix Anal. Appl..

[7]  Jérôme Idier,et al.  Algorithms for Nonnegative Matrix Factorization with the β-Divergence , 2010, Neural Computation.

[8]  D. Butnariu,et al.  Convergence of Bregman Projection Methods for Solving Consistent Convex Feasibility Problems in Reflexive Banach Spaces , 1997 .

[9]  Marc Teboulle,et al.  Convergence Analysis of a Proximal-Like Minimization Algorithm Using Bregman Functions , 1993, SIAM J. Optim..

[10]  Frank Deutsch,et al.  The Method of Alternating Orthogonal Projections , 1992 .

[11]  Peter Richtárik,et al.  Stochastic Reformulations of Linear Systems: Algorithms and Convergence Theory , 2017, SIAM J. Matrix Anal. Appl..

[12]  Per-Gunnar Martinsson,et al.  Randomized Numerical Linear Algebra: Foundations & Algorithms , 2020, ArXiv.

[13]  Marco Cuturi,et al.  Computational Optimal Transport: With Applications to Data Science , 2019 .

[14]  Alfredo N. Iusem,et al.  Iterative Methods of Solving Stochastic Convex Feasibility Problems and Applications , 2000, Comput. Optim. Appl..

[15]  P. L. Combettes.  The foundations of set theoretic estimation , 1993 .

[16]  Nicolas Papadakis,et al.  Regularized Optimal Transport and the Rot Mover's Distance , 2016, J. Mach. Learn. Res..

[17]  Philip A. Knight,et al.  The Sinkhorn-Knopp Algorithm: Convergence and Applications , 2008, SIAM J. Matrix Anal. Appl..

[18]  C. Castaing,et al.  Convex analysis and measurable multifunctions , 1977 .

[19]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[20]  Patrick L. Combettes,et al.  Convex set theoretic image recovery by extrapolated iterations of parallel subgradient projections , 1997, IEEE Trans. Image Process..

[21]  Heinz H. Bauschke,et al.  Bregman distances and Chebyshev sets , 2007, J. Approx. Theory.

[22]  Frank Nielsen,et al.  Tsallis Regularized Optimal Transport and Ecological Inference , 2016, AAAI.

[23]  L. Rüschendorf.  Convergence of the iterative proportional fitting procedure , 1995 .

[24]  Jason Altschuler,et al.  Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration , 2017, NIPS.

[25]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[26]  Y. Censor,et al.  An iterative row-action method for interval convex programming , 1981 .

[27]  Babak Hassibi,et al.  Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization , 2018, ICLR.

[28]  R. Durrett.  Probability: Theory and Examples , 1993 .

[29]  Alfredo N. Iusem.  On Dual Convergence and the Rate of Primal Convergence of Bregman's Convex Programming Method , 1991, SIAM J. Optim..

[30]  Peter Richtárik,et al.  Randomized Projection Methods for Convex Feasibility: Conditioning and Convergence Rates , 2019, SIAM J. Optim..

[31]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[32]  Heinz H. Bauschke,et al.  Legendre functions and the method of random Bregman projections , 1997 .

[33]  D. Butnariu,et al.  Strong convergence of expected-projection methods in hilbert spaces , 1995 .

[34]  Deanna Needell,et al.  Adaptive Sketch-and-Project Methods for Solving Linear Systems , 2019, ArXiv.

[35]  Heinz H. Bauschke,et al.  ESSENTIAL SMOOTHNESS, ESSENTIAL STRICT CONVEXITY, AND LEGENDRE FUNCTIONS IN BANACH SPACES , 2001 .

[36]  Gabriel Peyré,et al.  Iterative Bregman Projections for Regularized Transportation Problems , 2014, SIAM J. Sci. Comput..

[37]  D. R. Luke,et al.  Random Function Iterations for Consistent Stochastic Feasibility , 2018, Numerical Functional Analysis and Optimization.

[38]  Y. Censor,et al.  Parallel Optimization:theory , 1997 .

[39]  Angelia Nedic,et al.  Random projection algorithms for convex set intersection problems , 2010, 49th IEEE Conference on Decision and Control (CDC).

[40]  H. Robbins,et al.  A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications , 1985 .

[41]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .