Lower and upper bounds on the minimum mean-square error in composite source signal estimation

The performance of a minimum mean-square error (MMSE) estimator for the output signal from a composite source model (CSM), which has been degraded by statistically independent additive noise, is analyzed for a wide class of discrete-time and continuous-time models. In both cases, the MMSE is decomposed into the MMSE of the estimator, which is informed of the exact states of the signal and noise, and an additional error term. This term is tightly upper and lower bounded. The bounds for the discrete-time signals are developed using distribution tilting and Shannon's lower bound on the probability of a random variable exceeding a given threshold. The analysis for the continuous-time signal is performed using Duncan's theorem. The bounds in this case are developed by applying the data processing theorem to sampled versions of the state process and its estimate, and by using Fano's inequality. The bounds in both cases are explicitly calculated for CSMs with Gaussian subsources. For causal estimation, these bounds approach zero harmonically as the duration of the observed signals approaches infinity. >

[1]  Robert M. Gray,et al.  Probability, Random Processes, And Ergodic Properties , 1987 .

[2]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[3]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[4]  Robert M. Gray,et al.  Toeplitz And Circulant Matrices , 1977 .

[5]  D. Magill.  Optimal adaptive estimation of sampled stochastic processes , 1965 .

[6]  Demetrios G. Lainiotis,et al.  A class of upper bounds on probability of error for multihypotheses pattern recognition (Corresp.) , 1969, IEEE Trans. Inf. Theory.

[7]  Robert M. Gray,et al.  Rate-distortion speech coding with a minimum discrimination information distortion measure , 1981, IEEE Trans. Inf. Theory.

[8]  L. Merakos,et al.  Comments and corrections to 'New convergence bounds for Bayes estimators' by D. Kazakos , 1983, IEEE Trans. Inf. Theory.

[9]  Robert M. Gray,et al.  A unified approach for encoding clean and noisy sources by means of waveform and autoregressive model vector quantization , 1988, IEEE Trans. Inf. Theory.

[10]  R. Gray.  Entropy and Information Theory , 1990, Springer New York.

[11]  Amiel Feinstein,et al.  Information and information stability of random variables and processes , 1964 .

[12]  T. T. Kadota,et al.  Capacity of a continuous memoryless channel with feedback , 1971, IEEE Trans. Inf. Theory.

[13]  R. Gray,et al.  Distortion measures for speech processing , 1980 .

[14]  Yariv Ephraim,et al.  A Bayesian estimation approach for speech enhancement using hidden Markov models , 1992, IEEE Trans. Signal Process..

[15]  Demetrios Kazakos.  New convergence bounds for Bayes estimators , 1981, IEEE Trans. Inf. Theory.

[16]  Jack K. Wolf,et al.  Transmission of noisy information to a noisy receiver with minimum distortion , 1970, IEEE Trans. Inf. Theory.

[17]  Robert N. McDonough,et al.  Detection of signals in noise , 1971 .

[18]  Louis A. Liporace,et al.  Variance of Bayes estimates , 1971, IEEE Trans. Inf. Theory.

[19]  A. D. Wyner,et al.  On the Asymptotic Distribution of a Certain Functional of the Wiener Process , 1969 .

[20]  D. Lainiotis,et al.  Recursive algorithm for the calculation of the adaptive Kalman filter weighting coefficients , 1969 .

[21]  D. Lainiotis.  A class of upper-bounds on probability of error for multi-hypotheses pattern recognition , 1969 .

[22]  D. A. Bell,et al.  Information Theory and Reliable Communication , 1969 .

[23]  D. Lainiotis.  Optimal adaptive estimation: Structure and parameter adaption , 1971 .

[24]  A. Gualtierotti.  H. L. Van Trees, Detection, Estimation, and Modulation Theory, , 1976 .

[25]  U. Grenander,et al.  Toeplitz Forms And Their Applications , 1958 .

[26]  T. Duncan.  ON THE CALCULATION OF MUTUAL INFORMATION , 1970 .

[27]  J. Doob.  Stochastic processes , 1953 .

[28]  Biing-Hwang Juang,et al.  On the application of hidden Markov models for enhancing noisy speech , 1989, IEEE Trans. Acoust. Speech Signal Process..

[29]  D. Lainiotis.  Optimal adaptive estimation: Structure and parameter adaptation , 1970 .

[30]  Thomas Kailath,et al.  A general likelihood-ratio formula for random signals in Gaussian noise , 1969, IEEE Trans. Inf. Theory.