Acceleration of sequential subspace optimization in Banach spaces by orthogonal search directions

Abstract A standard solution technique for linear operator equations of first kind is the Landweber scheme which is an iterative method that uses the negative gradient of the current residual as search direction, which is also called the Landweber direction. Though this method proves to be stable with respect to noisy data, it is known to be numerically slow for problems in Hilbert spaces and this behavior shows to be even worse in some Banach space settings. This is why the idea came up to use several search directions instead of the Landweber direction only which has led to the development of Sequential Subspace Optimization (SESOP) methods. This idea is related to the famous Conjugate Gradient (CG) techniques that are known to be amongst the most effective methods to solve linear equations in Hilbert spaces. Since CG methods decisively make use of the inner product structure, they have been inherently restricted to Hilbert spaces so far. SESOP methods in Banach spaces do not share the conjugacy property with CG methods. In this article we use the concept of generalized orthogonality in Banach spaces and apply metric projections to orthogonalize the current Landweber direction with respect to the search space of the last iteration. This leads to an accelerated SESOP method which is confirmed by various numerical experiments. Moreover, in Hilbert spaces our method coincides with the Conjugate Gradient Normal Error (CGNE) or Craig’s method applied to the normal equation. We prove weak convergence to the exact solution. Furthermore we perform a couple of numerical tests on a linear problem involving a random matrix and on the problem of 2D computerized tomography where we use different l p -spaces. In all experiments the orthogonalization of the search space shows superior convergence properties compared to standard SESOP. This especially holds for p close to 1. Letting p → 2 the more we recover the conjugacy property for the search directions and the more the convergence behaves independently of the size of the search space.

[1]  Neculai Andrei.  Another nonlinear conjugate gradient algorithm for unconstrained optimization , 2009, Optim. Methods Softw..

[2]  A. Nemirovskii.  The regularizing properties of the adjoint gradient method in ill-posed problems , 1987 .

[3]  P. Hansen.  Discrete Inverse Problems: Insight and Algorithms , 2010 .

[4]  C. Lanczos.  Solution of Systems of Linear Equations by Minimized Iterations1 , 1952 .

[5]  E. Zeidler.  Nonlinear functional analysis and its applications , 1988 .

[6]  R. C. James.  Orthogonality in normed linear spaces , 1945 .

[7]  M. Day.  Uniform Convexity in Factor and Conjugate Spaces , 1944 .

[8]  J. A. Clarkson.  Uniformly convex spaces , 1936 .

[9]  Tal Schuster,et al.  Nonlinear iterative methods for linear ill-posed problems in Banach spaces , 2006 .

[10]  J. Schwartz,et al.  Linear Operators. Part I: General Theory. , 1960 .

[11]  F. Natterer.  The Mathematics of Computerized Tomography , 1986 .

[12]  Joseph Muscat.  Functional Analysis: An Introduction to Metric Spaces, Hilbert Spaces, and Banach Algebras , 2014 .

[13]  T. Manteuffel,et al.  A taxonomy for conjugate gradient methods , 1990 .

[14]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[15]  Gene H. Golub,et al.  Matrix computations , 1983 .

[16]  T. Schuster,et al.  Sequential subspace optimization for nonlinear inverse problems , 2016, 1602.06781.

[17]  M. Schechter.  Principles of Functional Analysis , 1971 .

[18]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[19]  R. Siezen,et al.  others , 1999, Microbial Biotechnology.

[20]  Anthony T. Chronopoulos,et al.  Iterative methods for nonlinear operator equations , 1992 .

[21]  M. Hanke,et al.  A convergence analysis of the Landweber iteration for nonlinear ill-posed problems , 1995 .

[22]  Dennis F. Cudia.  The geometry of Banach spaces , 1964 .

[23]  On the Geometry of Abstract Vector Spaces , 1934 .

[24]  A. Kirsch.  An Introduction to the Mathematical Theory of Inverse Problems , 1996, Applied Mathematical Sciences.

[25]  Y. Censor,et al.  Parallel Optimization:theory , 1997 .

[26]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[27]  C. E. Chidume,et al.  Geometric Properties of Banach Spaces and Nonlinear Iterations , 2009 .

[28]  F. Schöpfer,et al.  Fast regularizing sequential subspace optimization in Banach spaces , 2008 .

[29]  I. Ciorǎnescu.  Geometry of banach spaces, duality mappings, and nonlinear problems , 1990 .

[30]  Garrett Birkhoff,et al.  Orthogonality in linear metric spaces , 1935 .

[31]  Barbara Kaltenbacher,et al.  Regularization Methods in Banach Spaces , 2012, Radon Series on Computational and Applied Mathematics.

[32]  Frank Natterer,et al.  Mathematical methods in image reconstruction , 2001, SIAM monographs on mathematical modeling and computation.

[33]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[34]  I. Loris.  On the performance of algorithms for the minimization of ℓ1-penalized functionals , 2007, 0710.4082.

[35]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[36]  Ya-Xiang Yuan,et al.  A Subspace Study on Conjugate Gradient Algorithms , 1995 .

[37]  E. J. Craig.  The N‐Step Iteration Procedures , 1955 .

[38]  F. Schöpfer,et al.  An iterative regularization method for the solution of the split feasibility problem in Banach spaces , 2008 .

[39]  Y. Alber,et al.  James orthogonality and orthogonal decompositions of Banach spaces , 2005 .

[40]  F. Sch.  Iterative Regularization Methods for the Solution of the Split Feasibility Problem in Banach Spaces , 2007 .

[41]  M. Hanke.  Conjugate gradient type methods for ill-posed problems , 1995 .

[42]  Joram Lindenstrauss.  Classical Banach Spaces II: Function Spaces , 1979 .

[43]  Y. Censor,et al.  Parallel Optimization: Theory, Algorithms, and Applications , 1997 .

[44]  Gene H. Golub,et al.  Some History of the Conjugate Gradient and Lanczos Algorithms: 1948-1976 , 1989, SIAM Rev..

[45]  Alfred K. Louis,et al.  Metric and Bregman projections onto affine subspaces and their computation via sequential subspace optimization methods , 2008 .

[46]  Joram Lindenstrauss.  On the modulus of smoothness and divergent series in Banach spaces. , 1963 .

[47]  Zongben Xu,et al.  Characteristic inequalities of uniformly convex and uniformly smooth Banach spaces , 1991 .