Linear Regression With Gaussian Model Uncertainty: Algorithms and Bounds

In this paper, we consider the problem of estimating an unknown deterministic parameter vector in a linear regression model with random Gaussian uncertainty in the mixing matrix. We prove that the maximum-likelihood (ML) estimator is a (de)regularized least squares estimator and develop three alternative approaches for finding the regularization parameter that maximizes the likelihood. We analyze the performance using the Cramer-Rao bound (CRB) on the mean squared error, and show that the degradation in performance due the uncertainty is not as severe as may be expected. Next, we address the problem again assuming that the variances of the noise and the elements in the model matrix are unknown and derive the associated CRB and ML estimator. We compare our methods to known results on linear regression in the error in variables (EIV) model. We discuss the similarity between these two competing approaches, and provide a thorough comparison that sheds light on their theoretical and practical differences.

[1]  H. V. Trees,et al.  Exploring Estimator BiasVariance Tradeoffs Using the Uniform CR Bound , 2007 .

[2]  Yonina C. Eldar,et al.  Blind Minimax Estimation , 2007, IEEE Transactions on Information Theory.

[3]  Alfred O. Hero,et al.  Exploring estimator bias-variance tradeoffs using the uniform CR bound , 1996, IEEE Trans. Signal Process..

[4]  Ami Wiesel,et al.  Maximum Likelihood Estimation in Random Linear Models: Generalizations and Performance Analysis , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[5]  Jean-Jacques Fuchs,et al.  A New Approach to Variable Selection Using the TLS Approach , 2007, IEEE Transactions on Signal Processing.

[6]  BoydDepartment,et al.  Robust Solutions to l 1 , l 2 , and l 1 Uncertain LinearApproximation Problems using Convex Optimization 1 , 2007 .

[7]  Amir Beck,et al.  On the Solution of the Tikhonov Regularization of the Total Least Squares Problem , 2006, SIAM J. Optim..

[8]  Marc Teboulle,et al.  Finding a Global Optimal Solution for a Quadratically Constrained Fractional Quadratic Problem with Applications to the Regularized Total Least Squares , 2006, SIAM J. Matrix Anal. Appl..

[9]  G. Golub,et al.  Parameter Estimation in the Presence of Bounded Data Uncertainties , 1998, SIAM J. Matrix Anal. Appl..

[10]  C. Stein,et al.  Estimation with Quadratic Loss , 1992 .

[11]  Yonina C. Eldar.  Minimax estimation of deterministic parameters in linear models with a random model matrix , 2006, IEEE Transactions on Signal Processing.

[12]  A. Beck,et al.  Hidden convexity based near maximum-likelihood CDMA detection , 2005, IEEE 6th Workshop on Signal Processing Advances in Wireless Communications, 2005..

[13]  R. W. Miller,et al.  A modified Cramér-Rao bound and its applications (Corresp.) , 1978, IEEE Trans. Inf. Theory.

[14]  Ami Wiesel,et al.  Maximum likelihood estimation in linear models with a Gaussian model matrix , 2006, IEEE Signal Processing Letters.

[15]  Umberto Mengali,et al.  The modified Cramer-Rao bound in vector parameter estimation , 1998, IEEE Trans. Commun..

[16]  S. Kay.  Fundamentals of statistical signal processing: estimation theory , 1993 .

[17]  Fulvio Gini,et al.  On the use of Cramer-Rao-like bounds in the presence of random nuisance parameters , 2000, IEEE Trans. Commun..

[18]  Garng M. Huang,et al.  Iterative maximum-likelihood sequence estimation for space-time coded systems , 2001, IEEE Trans. Commun..

[19]  Yonina C. Eldar,et al.  Robust mean-squared error estimation in the presence of model uncertainties , 2005, IEEE Transactions on Signal Processing.

[20]  Laurent El Ghaoui,et al.  Robust Solutions to Least-Squares Problems with Uncertain Data , 1997, SIAM J. Matrix Anal. Appl..

[21]  Gene H. Golub,et al.  Tikhonov Regularization and Total Least Squares , 1999, SIAM J. Matrix Anal. Appl..

[22]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[23]  M. Levin.  Estimation of a system pulse transfer function in the presence of noise , 1964 .

[24]  P. Stoica,et al.  On nonexistence of the maximum likelihood estimate in blind multichannel identification , 2005, IEEE Signal Process. Mag..

[25]  Hagit Messer,et al.  A Barankin-type lower bound on the estimation error of a hybrid parameter vector , 1997, IEEE Trans. Inf. Theory.

[26]  Sabine Van Huffel,et al.  Total least squares problem - computational aspects and analysis , 1991, Frontiers in applied mathematics.

[27]  Arie Yeredor,et al.  The joint MAP-ML criterion and its relation to ML and to extended least-squares , 2000, IEEE Trans. Signal Process..

[28]  A. Wald.  The Fitting of Straight Lines if Both Variables are Subject to Error , 1940 .

[29]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[30]  Stephen P. Boyd,et al.  Robust solutions to l/sub 1/, l/sub 2/, and l/sub /spl infin// uncertain linear approximation problems using convex optimization , 1998, Proceedings of the 1998 American Control Conference. ACC (IEEE Cat. No.98CH36207).

[31]  M. Sion.  On general minimax theorems , 1958 .

[32]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .

[33]  Roger S. Cheng,et al.  Iterative EM receiver for space-time coded systems in MIMO frequency-selective fading channels with channel gain and order estimation , 2004, IEEE Transactions on Wireless Communications.

[34]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.