Minimum Entropy of Error Principle in Estimation

Abstract The principle of minimum error entropy estimation as found in the work of Weidemann and Stear is reformulated as a problem of finding optimum locations of probability densities in a given mixture such that the resulting (differential) entropy is minimized. New results concerning the entropy lower bound are derived. Continuity of the entropy and attaining the minimum entropy are proved in the case where the mixture is finite. Some other examples and situations, in particular that of symmetric unimodal densities, are studied in more detail.

[1]  L. L. Campbell.  Characterization of Entropy of Probability Distributions on the Real Line , 1972, Inf. Control..

[2]  Thomas J. Goblick,et al.  Theoretical limitations on the transmission of data from analog sources , 1965, IEEE Trans. Inf. Theory.

[3]  Edwin B. Stear,et al.  Entropy analysis of estimating systems , 1970, IEEE Trans. Inf. Theory.

[4]  David G. Messerschmitt,et al.  Quantizing for maximum output entropy (Corresp.) , 1971, IEEE Trans. Inf. Theory.

[5]  Lars-Erik Persson,et al.  Some properties of generalized exponential entropies with applications to data compression , 1992, Inf. Sci..

[6]  J. N. Kapur,et al.  Entropy optimization principles with applications , 1992 .

[7]  L. Campbell,et al.  Exponential entropy as a measure of extent of a distribution , 1966 .

[8]  Edwin B. Stear,et al.  Entropy Analysis of Parameter Estimation , 1969, Inf. Control..

[9]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[10]  I. Vajda.  Theory of statistical inference and information , 1989 .

[11]  Takashi Soeda,et al.  An Application of the Information Theory to Estimation Problems , 1976, Inf. Control..

[12]  J. Galdos,et al.  Information and distortion in reduced-order filter design , 1977, IEEE Trans. Inf. Theory.

[13]  Henry L. Weidemann.  Entropy Analysis of Feedback Control Systems1 1The research for this paper was supported in part by funds from the United States Air Force Office of Scientific Research under AFOSR Grant 699-67. , 1969 .

[14]  Sadao Ikeda.  Continuity and characterization of Shannon-Wiener information measure for continuous probability distributions , 1959 .

[15]  R. Gray.  Source Coding Theory , 1989 .