Quasi-Nonexpansive Iterations on the Affine Hull of Orbits: From Mann's Mean Value Algorithm to Inertial Methods

Fixed point iterations play a central role in the design and the analysis of a large number of optimization algorithms. We study a new iterative scheme in which the update is obtained by applying a composition of quasi-nonexpansive operators to a point in the affine hull of the orbit generated up to the current iterate. This investigation unifies several algorithmic constructs, including Mann's mean value method, inertial methods, and multilayer memoryless methods. It also provides a framework for the development of new algorithms, such as those we propose for solving monotone inclusion and minimization problems.

[1]  Isao Yamada,et al.  The Adaptive Projected Subgradient Method Constrained by Families of Quasi-nonexpansive Mappings and Its Application to Online Learning , 2010, SIAM J. Optim..

[2]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[3]  Otto Toeplitz,et al.  Über allgemeine lineare Mittelbildungen. , 1911 .

[4]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[5]  Patrick L. Combettes,et al.  Stochastic Quasi-Fejér Block-Coordinate Fixed Point Iterations with Random Sweeping , 2014 .

[6]  K. Knopp.  Theory and Application of Infinite Series , 1990 .

[7]  Jonathan M. Borwein,et al.  Convergence Rate Analysis for Averaged Fixed Point Iterations in Common Fixed Point Problems , 2015, SIAM J. Optim..

[8]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[9]  Boris Polyak.  Minimization of unsmooth functionals , 1969 .

[10]  W. R. Mann,et al.  Mean value methods in iteration , 1953 .

[11]  W. G. Dotson,et al.  On the Mann iterative process , 1970 .

[12]  P. L. Combettes,et al.  Quasi-Fejérian Analysis of Some Optimization Algorithms , 2001 .

[13]  Dirk A. Lorenz,et al.  An Inertial Forward-Backward Algorithm for Monotone Inclusions , 2014, Journal of Mathematical Imaging and Vision.

[14]  C. Byrne.  Iterative Optimization in Inverse Problems , 2014 .

[15]  C. W. Groetsch,et al.  A Note on Segmenting Mann Iterates , 1972 .

[16]  Felipe Alvarez,et al.  On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces , 2000, SIAM J. Control. Optim..

[17]  Philipp Kügler,et al.  Mean value iterations for nonlinear elliptic Cauchy problems , 2003, Numerische Mathematik.

[18]  P. L. Combettes,et al.  Compositions and convex combinations of averaged nonexpansive operators , 2014, 1407.5100.

[19]  P. L. Combettes,et al.  Generalized Mann iterates for constructing fixed points in Hilbert spaces , 2002 .

[20]  Georges Bouligand,et al.  Leçons sur les Séries Divergentes , 2009 .

[21]  Curtis Outlaw,et al.  Averaging iteration in a Banach space , 1969 .

[22]  P. L. Combettes.  Iterative construction of the resolvent of a sum of maximal monotone operators , 2009 .

[23]  Patrick L. Combettes,et al.  A Parallel Splitting Method for Coupled Monotone Inclusions , 2009, SIAM J. Control. Optim..

[24]  L. Fejér,et al.  Untersuchungen über Fouriersche Reihen , 1903 .

[25]  J. B. Diaz,et al.  On the structure of the set of subsequential limit points of successive approximations , 1967 .

[26]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[27]  Damek Davis,et al.  Convergence Rate Analysis of Primal-Dual Splitting Schemes , 2014, SIAM J. Optim..

[28]  Felipe Alvarez,et al.  Weak Convergence of a Relaxed and Inertial Hybrid Projection-Proximal Point Algorithm for Maximal Monotone Operators in Hilbert Space , 2003, SIAM J. Optim..

[29]  Patrick L. Combettes,et al.  Consistent learning by composite proximal thresholding , 2015, Mathematical Programming.

[30]  Diana Barro,et al.  Combining stochastic programming and optimal control to decompose multistage stochastic optimization problems , 2016, OR Spectr..

[31]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[32]  Claude Brezinski,et al.  Nonlinear hybrid procedures and fixed point iterations , 1998 .

[33]  K. L. Singh,et al.  FIXED POINT ITERATIONS USING INFINITE MATRICES , 1979 .

[34]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[35]  Averaging to improve convergence of iterative processes , 1979 .

[36]  Boris Polyak.  Gradient methods for the minimisation of functionals , 1963 .

[37]  J. Moreau.  Fonctions convexes duales et points proximaux dans un espace hilbertien , 1962 .

[38]  H. Attouch,et al.  An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping , 2001 .

[39]  Hedy Attouch,et al.  The Rate of Convergence of Nesterov's Accelerated Forward-Backward Method is Actually Faster Than 1/k2 , 2015, SIAM J. Optim..

[40]  Radu Ioan Bot,et al.  Inertial Douglas-Rachford splitting for monotone inclusion problems , 2014, Appl. Math. Comput..

[41]  Jonathan M. Borwein,et al.  Fixed point iterations for real functions , 1991 .

[42]  H. Attouch,et al.  THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM , 2000 .