Structured Total Maximum Likelihood: An Alternative to Structured Total Least Squares

Linear inverse problems with uncertain measurement matrices appear in many different applications. One of the standard techniques for solving such problems is the total least squares (TLS) method. Recently, an alternative approach has been suggested, based on maximizing an appropriate likelihood function assuming that the measurement matrix consists of random Gaussian variables. We refer to this technique as the total maximum likelihood (TML) method. Here we extend this strategy to the case in which the measurement matrix is structured so that the perturbations are not arbitrary but rather follow a fixed pattern. The resulting estimate is referred to as the structured TML (STML). As we show, the STML can be viewed as a regularized version of the structured TLS (STLS) approach in which the regularization consists of a logarithmic penalty. In contrast to the STLS solution, the STML always exists. Furthermore, its performance in practice tends to be superior to that of the STLS and competitive to other regularized solvers, as we illustrate via several examples. We also consider a few interesting special cases in which the STML can be computed efficiently either by reducing it into a one-dimensional problem regardless of the problem size or by a decomposition via a discrete Fourier transform.

[1]  David L. Phillips,et al.  A Technique for the Numerical Solution of Certain Integral Equations of the First Kind , 1962, JACM.

[2]  M. Aoki,et al.  On a priori error estimates of some identification methods , 1970 .

[3]  B. R. Hunt,et al.  Digital Image Restoration , 1977 .

[4]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[5]  Charles M. Rader,et al.  Fast transforms: Algorithms, analyses, applications , 1984 .

[6]  Wayne A. Fuller,et al.  Measurement Error Models , 1988 .

[7]  H. Zha,et al.  The restricted total least squares problem: formulation, algorithm, and properties , 1991 .

[8]  Jerry M. Mendel,et al.  The constrained total least squares technique and its applications to harmonic superresolution , 1991, IEEE Trans. Signal Process..

[9]  Sabine Van Huffel,et al.  Total least squares problem - computational aspects and analysis , 1991, Frontiers in applied mathematics.

[10]  C. Stein,et al.  Estimation with Quadratic Loss , 1992 .

[11]  J. J. Moré Generalizations of the trust region problem , 1993 .

[12]  Bart De Moor,et al.  Total least squares for affinely structured matrices and the noisy realization problem , 1994, IEEE Trans. Signal Process..

[13]  Sailes K. Sengijpta Fundamentals of Statistical Signal Processing: Estimation Theory , 1995 .

[14]  J. Navarro-Pedreño Numerical Methods for Least Squares Problems , 1996 .

[15]  J. Ben Rosen,et al.  Total Least Norm Formulation and Solution for Structured Problems , 1996, SIAM J. Matrix Anal. Appl..

[16]  Laurent El Ghaoui,et al.  Robust Solutions to Least-Squares Problems with Uncertain Data , 1997, SIAM J. Matrix Anal. Appl..

[17]  G. Golub,et al.  Parameter Estimation in the Presence of Bounded Data Uncertainties , 1998, SIAM J. Matrix Anal. Appl..

[18]  Gene H. Golub,et al.  Tikhonov Regularization and Total Least Squares , 1999, SIAM J. Matrix Anal. Appl..

[19]  Sabine Van Huffel,et al.  Fast Structured Total Least Squares Algorithm for Solving the Basic Deconvolution Problem , 2000, SIAM J. Matrix Anal. Appl..

[20]  D. Bertsekas 6.253 Convex Analysis and Optimization, Spring 2010 , 2004 .

[21]  Sabine Van Huffel,et al.  Analysis of the Structured Total Least Squares Problem for Hankel/Toeplitz Matrices , 2004, Numerical Algorithms.

[22]  Rosemary A. Renaut,et al.  Efficient Algorithms for Solution of Regularized Total Least Squares , 2005, SIAM J. Matrix Anal. Appl..

[23]  G. Golub,et al.  Regularized Total Least Squares Based on Quadratic Eigenvalue Problem Solvers , 2004 .

[24]  Yonina C. Eldar,et al.  Robust mean-squared error estimation in the presence of model uncertainties , 2005, IEEE Transactions on Signal Processing.

[25]  Per Christian Hansen,et al.  REGULARIZATION TOOLS: A Matlab package for analysis and solution of discrete ill-posed problems , 1994, Numerical Algorithms.

[26]  Amir Beck,et al.  A Global Solution for the Structured Total Least Squares Problem with Block Circulant Matrices , 2005, SIAM J. Matrix Anal. Appl..

[27]  Sabine Van Huffel,et al.  Block-Toeplitz/Hankel Structured Total Least Squares , 2005, SIAM J. Matrix Anal. Appl..

[28]  Ami Wiesel,et al.  Maximum likelihood estimation in linear models with a Gaussian model matrix , 2006, IEEE Signal Processing Letters.

[29]  Amir Beck,et al.  On the Solution of the Tikhonov Regularization of the Total Least Squares Problem , 2006, SIAM J. Optim..

[30]  Yonina C. Eldar Minimax estimation of deterministic parameters in linear models with a random model matrix , 2006, IEEE Transactions on Signal Processing.

[31]  Yonina C. Eldar,et al.  Strong Duality in Nonconvex Quadratic Optimization with Two Quadratic Constraints , 2006, SIAM J. Optim..

[32]  Jianhong Shen,et al.  Deblurring images: Matrices, spectra, and filtering , 2007, Math. Comput..

[33]  Amir Beck,et al.  The matrix-restricted total least-squares problem , 2007, Signal Process..

[34]  Yonina C. Eldar,et al.  Blind Minimax Estimation , 2007, IEEE Transactions on Information Theory.

[35]  Gene H. Golub,et al.  Generalized cross-validation as a method for choosing a good ridge parameter , 1979, Milestones in Matrix Computation.

[36]  Sabine Van Huffel,et al.  Overview of total least-squares methods , 2007, Signal Process..

[37]  Gene H. Golub,et al.  An analysis of the total least squares problem , 1980, Milestones in Matrix Computation.

[38]  Yonina C. Eldar Rethinking Biased Estimation: Improving Maximum Likelihood and the Cramér-Rao Bound , 2008, Found. Trends Signal Process..

[39]  Yonina C. Eldar,et al.  Rethinking Biased Estimation , 2008 .

[40]  Ami Wiesel,et al.  Linear Regression With Gaussian Model Uncertainty: Algorithms and Bounds , 2008, IEEE Transactions on Signal Processing.

[41]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[42]  S. R. Simanca,et al.  On Circulant Matrices , 2012 .