Estimating functions of probability distributions from a finite set of samples.

We present estimators for entropy and other functions of a discrete probability distribution when the data is a finite sample drawn from that probability distribution. In particular, for the case when the probability distribution is a joint distribution, we present finite sample estimators for the mutual information, covariance, and chi-squared functions of that probability distribution.

[1]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[2]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[3]  Walter L. Smith.  Probability and Statistics , 1959, Nature.

[4]  S. Fomin,et al.  Measure, Lebesgue integrals, and Hilbert space , 1961 .

[5]  W. Rudin.  Principles of mathematical analysis , 1964 .

[6]  R. A. Silverman,et al.  Special functions and their applications , 1966 .

[7]  J. Powell.  Mathematical Methods in Physics , 1965 .

[8]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .

[9]  Peter K. F. Kuhfittig,et al.  Introduction to the Laplace transform , 1978 .

[10]  Richard W. Hamming,et al.  Coding and Information Theory , 1980 .

[11]  D. Rubin.  The Bayesian Bootstrap , 1981 .

[12]  E. T. Jaynes,et al.  Papers on probability, statistics and statistical physics , 1983 .

[13]  John Skilling,et al.  Maximum entropy method in image processing , 1984 .

[14]  Y. Tikochinsky,et al.  Consistent inference of probabilities for reproducible experiments , 1984 .

[15]  Satosi Watanabe,et al.  Pattern Recognition: Human and Mechanical , 1985 .

[16]  E. T. Jaynes,et al.  Maximum Entropy and Bayesian Methods in Applied Statistics: Monkeys, Kangaroos, and N , 1986 .

[17]  P. Grassberger.  Finite sample corrections to entropy and dimension estimates , 1988 .

[18]  J. Skilling.  Classic Maximum Entropy , 1989 .

[19]  Wentian Li,et al.  Mutual Information Functions of Natural Language Texts , 1989 .

[20]  Wentian Li.  Mutual information functions versus correlation functions , 1990 .

[21]  Tom Leonard,et al.  Bayesian Inference for a Covariance Matrix , 1992 .

[22]  D. R. Wolf,et al.  Alpha, Evidence, and the Entropic Prior , 1993 .

[23]  David H. Wolpert,et al.  Estimating Functions of Distributions from A Finite Set of Samples, Part 2: Bayes Estimators for Mutual Information, Chi-Squared, Covariance and other Statistics , 1994, comp-gas/9403002.