A Note on the W-S Lower Bound of the MEE Estimation

The minimum error entropy (MEE) estimation is concerned with the estimation of a certain random variable (unknown variable) based on another random variable (observation), so that the entropy of the estimation error is minimized. This estimation method may outperform the well-known minimum mean square error (MMSE) estimation especially for non-Gaussian situations. There is an important performance bound on the MEE estimation, namely the W-S lower bound, which is computed as the conditional entropy of the unknown variable given observation. Though it has been known in the literature for a considerable time, up to now there is little study on this performance bound. In this paper, we reexamine the W-S lower bound. Some basic properties of the W-S lower bound are presented, and the characterization of Gaussian distribution using the W-S lower bound is investigated.

[1]  Badong Chen,et al.  System Parameter Identification: Information Criteria and Algorithms , 2013 .

[2]  Ting-Li Chen,et al.  On the Minimum Entropy of a Mixture of Unimodal and Symmetric Distributions , 2008, IEEE Transactions on Information Theory.

[3]  I. Olkin,et al.  A Characterization of the Multivariate Normal Distribution , 1962 .

[4]  Deniz Erdogmus,et al.  Generalized information potential criterion for adaptive system training , 2002, IEEE Trans. Neural Networks.

[5]  Jose C. Principe,et al.  Information Theoretic Learning - Renyi's Entropy and Kernel Perspectives , 2010, Information Theoretic Learning.

[6]  P. R. Fisk,et al.  A NOTE ON A CHARACTERIZATION OF THE MULTIVARIATE NORMAL DISTRIBUTION , 1970 .

[7]  Antonín Otáhal.  Minimum entropy of error estimate for multi-dimensional parameter and finite-state-space observations , 1995, Kybernetika.

[8]  Edwin B. Stear,et al.  Entropy analysis of estimating systems , 1970, IEEE Trans. Inf. Theory.

[9]  Jesús López Fidalgo,et al.  Characterizing the General Multivariate Normal Distribution through the Conditional Distributions , 1997 .

[10]  Deniz Erdogmus,et al.  Entropy minimization for supervised digital communications channel equalization , 2002, IEEE Trans. Signal Process..

[11]  Nariyasu Minamide.  An Extension of the Entropy Theorem for Parameter Estimation , 1982, Inf. Control..

[12]  J.C. Principe,et al.  From linear adaptive filtering to nonlinear information processing - The design and analysis of information processing systems , 2006, IEEE Signal Processing Magazine.

[13]  Deniz Erdogmus,et al.  Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training , 2003, IEEE Trans. Signal Process..

[14]  Luc Pronzato,et al.  Minimum entropy estimation in semi parametric models , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[15]  Ming Zhang,et al.  On optimal estimations with minimum error entropy criterion , 2010, J. Frankl. Inst..

[16]  Gholamhossein Hamedani,et al.  On a recent characterization of the bivariate normal distribution , 1991 .

[17]  Takashi Soeda,et al.  An Application of the Information Theory to Estimation Problems , 1976, Inf. Control..

[18]  Badong Chen,et al.  On the Smoothed Minimum Error Entropy Criterion , 2012, Entropy.

[19]  Zengqi Sun,et al.  Stochastic Gradient Algorithm Under (h,φ)-Entropy Criterion , 2007 .

[20]  Martin Janzura,et al.  Minimum Entropy of Error Principle in Estimation , 1994, Inf. Sci..

[21]  Wolfgang Bischoff,et al.  Characterization of the multivariate normal distribution by conditional normal distributions , 1991 .

[22]  Ming Zhang,et al.  A new interpretation on the MMSE as a robust MEE criterion , 2010, Signal Process..

[23]  Badong Chen,et al.  Some Further Results on the Minimum Error Entropy Estimation , 2012, Entropy.

[24]  Paul Kalata,et al.  Linear prediction, filtering, and smoothing: An information-theoretic approach , 1979, Inf. Sci..

[25]  Deniz Erdogmus,et al.  An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems , 2002, IEEE Trans. Signal Process..