Bayes Estimate and Inference for Entropy and Information Index of Fit

This article defines a quantized entropy and develops Bayes estimates and inference for the entropy and a Kullback–Leibler information index of the model fit. We use a Dirichlet process prior for the unknown data-generating distribution with a maximum entropy candidate model as the expected distribution. This formulation produces prior and posterior distributions for the quantized entropy, the information index of fit, the moments, and the model parameters. The posterior mean of the quantized entropy provides a Bayes estimate of entropy under quadratic loss. The consistency of the Bayes estimates and the information index are shown. The implementation and the performances of the Bayes estimates are illustrated using data simulated from exponential, gamma, and lognormal distributions.

[1]  Emanuel Parzen,et al.  Quantile Probability and Statistical Data Modeling , 2004 .

[2]  D. Owen.  Handbook of Mathematical Functions with Formulas , 1965 .

[3]  Nader Ebrahimi,et al.  Testing exponentiality based on Kullback-Leibler information , 1992 .

[4]  Hiroshi Ohta,et al.  A Test for Normality Based on , 1989 .

[5]  Nader Ebrahimi,et al.  Ordering univariate distributions by entropy and variance , 1999 .

[6]  J. Bernardo.  Expected Information as Expected Utility , 1979 .

[7]  Arnold Zellner,et al.  Models, prior information, and Bayesian analysis☆ , 1996 .

[8]  D. Joanes,et al.  Bayesian estimation of Shannon's index of diversity , 1979 .

[9]  Nader Ebrahimi,et al.  Information distinguishability with application to analysis of failure data , 1995 .

[10]  N. Ebrahimi,et al.  Two measures of sample entropy , 1994 .

[11]  P. L. Novi Inverardi,et al.  MSE Comparison of Some Different Estimators of Entropy , 2003 .

[12]  Oldrich A Vasicek,et al.  A Test for Normality Based on Sample Entropy , 1976 .

[13]  Adrian Bowyer,et al.  Computing Dirichlet Tessellations , 1981, Comput. J..

[14]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[15]  Sangun Park.  A goodness-of-fit test for normality based on the sample entropy of order statistics , 1999 .

[16]  Arthur P. Dempster,et al.  The direct use of likelihood for significance testing , 1997, Stat. Comput..

[17]  Thomas B. Fomby,et al.  Applying maximum entropy to econometric problems , 1997 .

[18]  H. Ohta,et al.  A Test for Normality Based on Kullback—Leibler Information , 1989 .

[19]  P. Hall,et al.  On the estimation of entropy , 1993 .

[20]  Ehsan S. Soofi,et al.  Information measures for generalized gamma family , 2007 .

[21]  Henri Theil,et al.  The entropy of the maximum entropy distribution , 1980 .

[22]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[23]  Narayanaswamy Balakrishnan,et al.  Testing Exponentiality Based on Kullback-Leibler Information With Progressively Type-II Censored Data , 2007, IEEE Transactions on Reliability.

[24]  B. Clarke,et al.  On the overall sensitivity of the posterior distribution to its inputs , 1998 .

[25]  Testing whether lifetime distribution is decreasing uncertainty , 1997 .

[26]  Nader Ebrahimi.  Testing exponentiality of the residual life, based on dynamic Kullback-Leibler information , 1998 .

[27]  G. S. Mudholkar,et al.  An entropy characterization of the inverse Gaussian distribution and related goodness-of-fit test , 2002 .

[28]  Arnold Zellner,et al.  Bayesian Methods and Entropy in Economics and Econometrics , 1991 .

[29]  L. Lorne Campbell.  Averaging entropy , 1995, IEEE Trans. Inf. Theory.

[30]  L. Györfi,et al.  Nonparametric entropy estimation. An overview , 1997 .

[31]  E. Soofi,et al.  Information indices: unification and applications , 2002 .

[32]  D. Gokhale.  On entropy-based goodness-of-fit tests , 1983 .

[33]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[34]  N. Ebrahimi.  Testing for Uniformity of the Residual Life Time Based on Dynamic Kullback-Leibler Information , 2001 .

[35]  Ehsan S. Soofi,et al.  Computation of maximum entropy Dirichlet for modeling lifetime data , 2000 .

[36]  Edward C. van der Meulen,et al.  Entropy-Based Tests of Uniformity , 1981 .

[37]  Nicholas G. Polson,et al.  Diagnostic Measures for Model Criticism , 1996 .

[38]  Emanuele Taufer,et al.  ON ENTROPY BASED TESTS FOR EXPONENTIALITY , 2002 .

[39]  D. D. Waal.  Goodness of fit of the generalized extreme value distribution based on the Kullback-Leibler information , 1996 .

[40]  A. Yuan,et al.  An Information Criterion for Likelihood Selection , 1999, IEEE Trans. Inf. Theory.

[41]  Keeyoung Kim,et al.  Testing goodness-of-fit for Laplace distribution based on maximum entropy , 2006 .

[42]  Sangun Park,et al.  Testing exponentiality based on the Kullback-Leibler information with the type II censored data , 2005, IEEE Transactions on Reliability.

[43]  Ehsan S. Soofi,et al.  Executives' perceived environmental uncertainty shortly after 9/11 , 2009, Comput. Stat. Data Anal..

[44]  H. Akaike.  A new look at the statistical model identification , 1974 .

[45]  Sangun Park,et al.  Correcting moments for goodness of fit tests based on two entropy estimates , 2003 .

[46]  Milton Abramowitz,et al.  Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables , 1964 .

[47]  H. Joe.  Estimation of entropy and other functionals of a multivariate density , 1989 .

[48]  E. Soofi.  INFORMATION THEORETIC REGRESSION METHODS , 1997 .

[49]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[50]  Bertrand Clarke,et al.  Implications of Reference Priors for Prior Information and for Sample Size , 1996 .