Conjugate Gradient Methods for Toeplitz Systems

In this expository paper, we survey some of the latest developments in using preconditioned conjugate gradient methods for solving Toeplitz systems. One of the main results is that the complexity of solving a large class of $n$-by-$n$ Toeplitz systems is reduced to $O(n \log n)$ operations as compared to $O(n \log ^2 n)$ operations required by fast direct Toeplitz solvers. Different preconditioners proposed for Toeplitz systems are reviewed. Applications to Toeplitz-related systems arising from partial differential equations, queueing networks, signal and image processing, integral equations, and time series analysis are given.

[1]  J. Nagy,et al.  DISPLACEMENT PRECONDITIONER FOR TOEPLITZ LEAST SQUARES ITERATIONS , 1994 .

[2]  James G. Nagy,et al.  Iterative image restoration using approximate inverse preconditioning , 1996, IEEE Trans. Image Process..

[3]  Andrew E. Yagle.  New analogs of split algorithms for arbitrary Toeplitz-plus-Hankel matrices , 1991, IEEE Trans. Signal Process..

[4]  Majid Ahmadi,et al.  Digital Filtering in One and Two Dimensions: Design and Applications , 1989 .

[5]  Clyde Martin,et al.  Analytic and numerical aspects of the observation of the heat equation , 1987, 26th IEEE Conference on Decision and Control.

[6]  Raymond H. Chan,et al.  The Numerical Solution of the Biharmonic Equation by Conformal Mapping , 1997, SIAM J. Sci. Comput..

[7]  L. Trefethen.  Approximation theory and numerical linear algebra , 1990 .

[8]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[9]  G. Yule.  On a Method of Investigating Periodicities in Disturbed Series, with Special Reference to Wolfer's Sunspot Numbers , 1927 .

[10]  D. Sweet.  Fast Toeplitz orthogonalization , 1984 .

[11]  Paola Favati,et al.  On a Matrix Algebra Related to the Discrete Hartley Transform , 1993, SIAM J. Matrix Anal. Appl..

[12]  S. Holmgren,et al.  Iterative Solution Methods and Preconditioners for Block-Tridiagonal Systems of Equations , 1992, SIAM J. Matrix Anal. Appl..

[13]  I. Koltracht,et al.  Fast Transform Based Preconditioners for Toeplitz Equations , 1995, SIAM J. Matrix Anal. Appl..

[14]  Thomas Huckle,et al.  Circulant and Skewcirculant Matrices for Solving Toeplitz Matrix Problems , 1992, SIAM J. Matrix Anal. Appl..

[15]  Raymond H. Chan,et al.  SINE TRANSFORM BASED PRECONDITIONERS FOR SYMMETRIC TOEPLITZ SYSTEMS , 1996 .

[16]  A. Jain.  Fast inversion of banded Toeplitz matrices by circular decompositions , 1978 .

[17]  Å. Björck.  Least squares methods , 1990 .

[18]  Shalhav Zohar,et al.  The Solution of a Toeplitz Set of Linear Equations , 1974, JACM.

[19]  Stefano Serra,et al.  Preconditioning strategies for asymptotically ill-conditioned block Toeplitz systems , 1994 .

[20]  P. Anselone,et al.  Collectively Compact Operator Approximation Theory and Applications to Integral Equations , 1971 .

[21]  Guanrong Chen,et al.  Signal processing and systems theory , 1992 .

[22]  Raymond H. Chan,et al.  Circulant and skew-circulant preconditioners for skew-hermitian type Toeplitz systems , 1991 .

[23]  C.-C. Jay Kuo,et al.  Design and analysis of Toeplitz preconditioners , 1992, IEEE Trans. Signal Process..

[24]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[25]  S. T. Alexander,et al.  Adaptive Signal Processing: Theory and Applications , 1986 .

[26]  Otto Toeplitz,et al.  Zur Theorie der quadratischen und bilinearen Formen von unendlichvielen Veränderlichen , 1911 .

[27]  Martin H. Gutknecht,et al.  Stable row recurrences for the Padé table and generically superfast lookahead solvers for non-Hermitian Toeplitz systems , 1993 .

[28]  R. Chan.  Toeplitz Preconditioners for Toeplitz Systems with Nonnegative Generating Functions , 1991 .

[29]  Claude Guéguen,et al.  Linear prediction in the singular case and the stability of eigen models , 1981, ICASSP.

[30]  Thomas Huckle,et al.  Some Aspects of Circulant Preconditioners , 1993, SIAM J. Sci. Comput..

[31]  Roland W. Freund,et al.  A look-ahead Bareiss algorithm for general Toeplitz matrices , 1994 .

[32]  Eugene E. Tyrtyshnikov,et al.  Circulant preconditioners with unbounded inverses , 1995 .

[33]  G. Szegő,et al.  On the Eigen-Values of Certain Hermitian Forms , 1953 .

[34]  Thomas Huckle,et al.  Iterative methods for Toeplitz-like matrices , 1994 .

[35]  Stephen J. Wright,et al.  Parallel Algorithms for Banded Linear Systems , 1991, SIAM J. Sci. Comput..

[36]  J. Tukey,et al.  An algorithm for the machine calculation of complex Fourier series , 1965 .

[37]  David Y. Y. Yun,et al.  Fast Solution of Toeplitz Systems of Equations and Computation of Padé Approximants , 1980, J. Algorithms.

[38]  Raymond H. Chan,et al.  Fast Iterative Solvers for Toeplitz-Plus-Band Systems , 1993, SIAM J. Sci. Comput..

[39]  Philippe Delsarte,et al.  A generalization of the Levinson algorithm for Hermitian Toeplitz matrices with any rank profile , 1985, IEEE Trans. Acoust. Speech Signal Process..

[40]  C.-C. Jay Kuo,et al.  Preconditioned iterative methods for solving Toeplitz-plus-Hankel systems , 1993 .

[41]  R. Chan,et al.  Circulant preconditioners for complex Toeplitz matrices , 1993 .

[42]  Selim G. Akl,et al.  Design and analysis of parallel algorithms , 1985 .

[43]  I. Gustafsson.  A class of first order factorization methods , 1978 .

[44]  R. Chan,et al.  Optimal trigonometric preconditioners for elliptic problems and queueing problems , 1996 .

[45]  Raymond H. Chan,et al.  Fast Band-Toeplitz Preconditioners for Hermitian Toeplitz Systems , 1994, SIAM J. Sci. Comput..

[46]  Raymond H. Chan,et al.  Circulant preconditioners for Toeplitz matrices with piecewise continuous generating functions , 1992 .

[47]  W. F. Trench.  An Algorithm for the Inversion of Finite Toeplitz Matrices , 1964 .

[48]  T. Kailath,et al.  Fast Parallel Algorithms for QR and Triangular Factorization , 1987 .

[49]  Per Christian Hansen,et al.  A look-ahead Levinson algorithm for general Toeplitz systems , 1992, IEEE Trans. Signal Process..

[50]  V. A. Barker,et al.  Finite element solution of boundary value problems , 1984 .

[51]  Judea Pearl,et al.  On coding and filtering stationary signals by discrete Fourier transforms (Corresp.) , 1973, IEEE Trans. Inf. Theory.

[52]  Lina Hemmingsson.  Toeplitz Preconditioners with Block Structure for First-order PDEs , 1996, Numer. Linear Algebra Appl..

[53]  Curtis R. Vogel,et al.  Iterative Methods for Total Variation Denoising , 1996, SIAM J. Sci. Comput..

[54]  R. Chan,et al.  The circulant operator in the banach algebra of matrices , 1991 .

[55]  Michael K. Ng,et al.  Fast iterative methods for solving Toeplitz-plus-Hankel least squares problems. , 1994 .

[56]  Andrew F. Seila,et al.  Multivariate Estimation of Conditional Performance Measures in Regenerative Simulation , 1990 .

[57]  Stefano Serra Capizzano,et al.  Multigrid Methods for Symmetric Positive Definite Block Toeplitz Matrices with Nonnegative Generating Functions , 1996, SIAM J. Sci. Comput..

[58]  S. Thomas Alexander,et al.  Adaptive Signal Processing , 1986, Texts and Monographs in Computer Science.

[59]  Roland W. Freund,et al.  A Transpose-Free Quasi-Minimal Residual Algorithm for Non-Hermitian Linear Systems , 1993, SIAM J. Sci. Comput..

[60]  D. B. Preston.  Spectral Analysis and Time Series , 1983 .

[61]  R. P. Kendall,et al.  An Approximate Factorization Procedure for Solving Self-Adjoint Elliptic Difference Equations , 1968 .

[62]  P. Lafrance,et al.  Digital filters , 1974, Proceedings of the IEEE.

[63]  Dario Bini,et al.  A new preconditioner for the parallel solution of positive definite Toeplitz systems , 1990, SPAA '90.

[64]  A. Chan,et al.  Application of approximation theory methods to recursive digital filter design , 1982 .

[65]  Raymond H. Chan,et al.  Sine transform based preconditioners for elliptic problems , 1997, Numer. Linear Algebra Appl..

[66]  Raymond H. Chan,et al.  Circulant preconditioners for elliptic problems , 1992 .

[67]  Eugene E. Tyrtyshnikov,et al.  Which circulant preconditioner is better? , 1996, Math. Comput..

[68]  Thomas Huckle,et al.  Fast transforms for tridiagonal linear equations , 1994 .

[69]  George Carayannis,et al.  Fast recursive algorithms for a class of linear equations , 1982 .

[70]  Raymond H. Chan,et al.  Toeplitz preconditioners for Hermitian Toeplitz systems , 1993 .

[71]  P. Yip,et al.  Fast decimation-in-time algorithms for a family of discrete sine and cosine transforms , 1984 .

[72]  Richard A. Davis,et al.  Time Series: Theory and Methods (2nd ed.). , 1992 .

[73]  R. Mersereau,et al.  Iterative methods for image deblurring , 1990 .

[74]  R. Chan,et al.  Circulant preconditioners constructed from kernels , 1992 .

[75]  Andrew E. Yagle,et al.  Fast algorithms for close-to-Toeplitz-plus-Hankel systems and two-sided linear prediction , 1993, IEEE Trans. Signal Process..

[76]  Raymond H. Chan,et al.  The spectra of super-optimal circulant preconditioned Toeplitz systems , 1991 .

[77]  R. Chan,et al.  A Family of Block Preconditioners for Block Systems , 1992, SIAM J. Sci. Comput..

[78]  R. Hamming.  Digital filters (3rd ed.) , 1989 .

[79]  G. Golub,et al.  Iterative solution of linear systems , 1991, Acta Numerica.

[80]  P. Heywood.  Trigonometric Series , 1968, Nature.

[81]  J. Nagy,et al.  FFT-based preconditioners for Toeplitz-block least squares problems , 1993 .

[82]  D. Sweet,et al.  The use of pivoting to improve the numerical performance of algorithms for Toeplitz matrices , 1993 .

[83]  Robert J. Plemmons,et al.  FFT-based RLS in signal processing , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[84]  Werner Henkel.  An extended Berlekamp-Massey algorithm for the inversion of Toeplitz matrices , 1992, IEEE Trans. Commun..

[85]  Lina Hemmingsson,et al.  Analysis of Semi-Toeplitz Preconditioners for First-Order PDEs , 1996, SIAM J. Sci. Comput..

[86]  C.-C. Jay Kuo,et al.  A Minimum-Phase LU Factorization Preconditioner for Toeplitz Matrices , 1992, SIAM J. Sci. Comput..

[87]  Sverker Holmgren,et al.  Semicirculant Preconditioners for First-Order Partial Differential Equations , 1994, SIAM J. Sci. Comput..

[88]  G. Strang.  A proposal for toeplitz matrix calculations , 1986 .

[89]  Eugene E. Tyrtyshnikov,et al.  Optimal and Superoptimal Circulant Preconditioners , 1992, SIAM J. Matrix Anal. Appl..

[90]  Kurt Otto,et al.  Analysis of Preconditioners for Hyperbolic Partial Differential Equations , 1996 .

[91]  A. Buckley,et al.  On the Solution of Certain Skew Symmetric Linear Systems , 1977 .

[92]  Tinsu Pan,et al.  Preconditioning methods for improved convergence rates in iterative reconstructions , 1993, IEEE Trans. Medical Imaging.

[93]  E. E. Tyrtyshnikov.  Influence of matrix operations on the distribution of Eigenvalues and singular values of Toeplitz matrices , 1994 .

[94]  Raymond H. Chan,et al.  Toeplitz-Circulant Preconditioners for Toeplitz Systems and their Applications to Queueing Networks with Batch Arrivals , 1996, SIAM J. Sci. Comput..

[95]  F. Hoog.  A new algorithm for solving Toeplitz systems of equations , 1987 .

[96]  Raymond H. Chan,et al.  Circulant preconditioners for second order hyperbolic equations , 1992 .

[97]  M. Tismenetsky,et al.  A decomposition of Toeplitz matrices and optimal circulant preconditioning , 1991 .

[98]  Chris Chatfield,et al.  Introduction to Statistical Time Series. , 1976 .

[99]  L. Delves,et al.  Computational methods for integral equations: Frontmatter , 1985 .

[100]  Raymond H. Chan,et al.  Best-conditioned circulant preconditioners , 1995 .

[101]  B. Anderson,et al.  Asymptotically fast solution of toeplitz and related systems of linear equations , 1980 .

[102]  E. E. Tyrtyshnikov.  A unifying approach to some old and new theorems on distribution and clustering , 1996 .

[103]  S. Barnett,et al.  Inversion of Toeplitz Matrices which are not Strongly Non-singular , 1985 .

[104]  C. J. Kuot,et al.  ON THE SPECTRUM OF A FAMILY OF PRECONDITIONED BLOCK TOEPLITZ MATRICES , 2022 .

[105]  Giuseppe Fiorentino,et al.  C. G. preconditioning for Toeplitz matrices , 1993 .

[106]  Prem K. Kythe,et al.  Integral Equation Methods , 1998 .

[107]  Raymond H. Chan,et al.  Generalization of Strang's Preconditioner with Applications to Toeplitz Least Squares Problems , 1996, Numer. Linear Algebra Appl..

[108]  O. Axelsson,et al.  Finite element solution of boundary value problemes - theory and computation , 2001, Classics in applied mathematics.

[109]  Guanrong Chen,et al.  Signal Processing and Systems Theory: Selected Topics , 1992 .

[110]  Paul N. Swarztrauber,et al.  Multiprocessor FFTs , 1987, Parallel Comput..

[111]  Stefano Serra,et al.  Multigrid methods for toeplitz matrices , 1991 .

[112]  Christopher J. Zarowski,et al.  Schur algorithms for Hermitian Toeplitz, and Hankel matrices with singular leading principal submatrices , 1991, IEEE Trans. Signal Process..

[113]  J. Nagy,et al.  Circulant Preconditioned Toeplitz Least Squares Iterations , 1994, SIAM J. Matrix Anal. Appl..

[114]  George Cybenko,et al.  Fast toeplitz orthogonalization using inner decompositions , 1987 .

[115]  S. Lawrence Marple.  Fast algorithms for linear prediction and system identification filters with linear phase , 1983, ICASSP.

[116]  Toshikane Oda.  Moment analysis for traffic associated with Markovian queueing systems , 1991, IEEE Trans. Commun..

[117]  Stanley J. Reeves,et al.  Optimal regularization parameter estimation for image restoration , 1991, Electronic Imaging.

[118]  Leon H. Sibul.  Adaptive signal processing , 1987 .

[119]  Raymond H. Chan,et al.  Circulant integral operators as preconditioners for Wiener-Hopf equations , 1995 .

[120]  Frank M. Hsu,et al.  Least Square Estimation with Applications to Digital Signal Processing , 1985 .

[121]  N. Wiener.  The Wiener RMS (Root Mean Square) Error Criterion in Filter Design and Prediction , 1949 .

[122]  P. J. Sherman.  Circulant approximations of the inverses of Toeplitz matrices and related quantities with applications to stationary random processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[123]  C.-C. Jay Kuo,et al.  Spectral properties of preconditioned rational Toeplitz matrices , 1993 .

[124]  Fabio Di Benedetto,et al.  Preconditioning of Block Toeplitz Matrices by Sine Transforms , 1997, SIAM J. Sci. Comput..

[125]  Anil K. Jain.  Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[126]  R. Chan.  Iterative methods for overflow queuing models II , 1988 .

[127]  C. J. KUOi.  SPECTRAL PROPERTIES OF PRECONDITIONED RATIONAL TOEPLITZ MATRICES : THE NONSYMMETRIC CASE , .

[128]  C. W. Groetsch,et al.  The theory of Tikhonov regularization for Fredholm equations of the first kind , 1984 .

[129]  Xiao-Qing Jin,et al.  Hartley preconditioners for Toeplitz systems generated by positive continuous functions , 1994 .

[130]  B. Kiss,et al.  A preconditioned domain decomposition algorithm for the solution of the elliptic Neumann problem , 1992 .

[131]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[132]  Raymond H. Chan,et al.  Construction of preconditioners for Wiener-Hopf equations by operator splitting , 1995 .

[133]  U. Grenander,et al.  Toeplitz Forms And Their Applications , 1958 .

[134]  Roland W. Freund,et al.  Formally biorthogonal polynomials and a look-ahead Levinson algorithm for general Teoplitz systems , 1993 .

[135]  R. H. Chan.  The spectrum of a family of circulant preconditioned Toeplitz systems , 1989 .

[136]  E. Cheney.  Introduction to approximation theory , 1966 .

[137]  D. Lieberman,et al.  Fourier analysis , 2004, Journal of cataract and refractive surgery.

[138]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[139]  C.-C. Jay Kuo,et al.  Preconditioned iterative methods for solving Toeplitz-plus-Hankel systems , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[140]  R. Brent,et al.  QR factorization of Toeplitz matrices , 1986 .

[141]  James G. Nagy,et al.  Fast restoration of atmospherically blurred images , 1994, Optics & Photonics.

[142]  Martin Morf,et al.  Doubling algorithms for Toeplitz and related equations , 1980, ICASSP.

[143]  Steve Rogers,et al.  Adaptive Filter Theory , 1996 .

[144]  P. Tang.  A fast algorithm for linear complex Chebyshev approximations , 1988 .

[145]  N. Levinson.  The Wiener (Root Mean Square) Error Criterion in Filter Design and Prediction , 1946 .

[146]  James G. Nagy,et al.  Preconditioned iterative regularization for Ill-posed problems , 1992 .

[147]  James G. Nagy,et al.  Some fast Toeplitz least-squares algorithms , 1991, Optics & Photonics.

[148]  Raymond H. Chan,et al.  Preconditioners for Wiener--Hopf Equations with High-Order Quadrature Rules , 1997 .

[149]  T. Chan.  An Optimal Circulant Preconditioner for Toeplitz Systems , 1988 .

[150]  Sverker Holmgren,et al.  Semicirculant Solvers and Boundary Corrections for First-Order Partial Differential Equations , 1996, SIAM J. Sci. Comput..

[151]  I. Gohberg,et al.  Convolution Equations and Projection Methods for Their Solution , 1974 .

[152]  L. Marple.  A new autoregressive spectrum analysis algorithm , 1980 .

[153]  Xiao-Qing Jin,et al.  Band Toeplitz preconditioners for block Toeplitz systems , 1996 .

[154]  W. Gragg,et al.  Superfast solution of real positive definite toeplitz systems , 1988 .

[155]  Raymond H. Chan,et al.  Jackson's theorem and circulant preconditioned Toeplitz systems , 1992 .

[156]  Paul Concus,et al.  A modified direct preconditioner for indefinite symmetric Toeplitz systems , 1995, Numer. Linear Algebra Appl..

[157]  M. Hanke,et al.  Fast preconditioned conjugate gradient algorithms for Wiener-Hopf integral equations , 1994 .

[158]  L. Ljung,et al.  New inversion formulas for matrices classified in terms of their distance from Toeplitz matrices , 1979 .

[159]  G. Strang,et al.  Toeplitz equations by conjugate gradients with circulant preconditioner , 1989 .

[160]  James R. Bunch,et al.  Stability of Methods for Solving Toeplitz Systems of Equations , 1985 .

[161]  Thomas Kailath,et al.  Divide-and-conquer solutions of least-squares problems for matrices with displacement structure , 1991 .

[162]  C. Loan.  Computational Frameworks for the Fast Fourier Transform , 1992 .