Numerical Evaluation of Information- Theoretic Measures

Information theoretic measures are of paramount importance in Bayesian inference. Yet, their direct application in practice has been somewhat limited by the severe computational diiculties encountered in complex problems. In this paper we discuss implementation strategies for fast numerical computations of Entropies and Kullback-Leibler divergences that are relevant to Bayesian inference and design problems. We illustrate the methods proposed with examples in model diagnostics and information theoretic design.

[1]  D. Lindley.  On a Measure of the Information Provided by an Experiment , 1956 .

[2]  David Lindley,et al.  BINOMIAL SAMPLING SCHEMES AND THE CONCEPT OF INFORMATION , 1957 .

[3]  Alexander A. Lubischew.  On the Use of Discriminant Functions in Taxonomy , 1962 .

[4]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[5]  Arnold Zellner,et al.  An Introduction to Bayesian Inference in Econometrics. , 1974 .

[6]  D. Byar,et al.  Comparisons of placebo, pyridoxine, and topical thiotepa in preventing recurrence of stage I bladder cancer. , 1977, Urology.

[7]  A. F. M. Smith,et al.  Progress with numerical and graphical methods for practical Bayesian statistics , 1987 .

[8]  Adrian F. M. Smith,et al.  Bayesian computation via the gibbs sampler and related markov chain monte carlo methods (with discus , 1993 .

[9]  J. Geweke,et al.  Bayesian Inference in Econometric Models Using Monte Carlo Integration , 1989 .

[10]  B. Carlin.  An Expected Utility Approach to Influence Diagnostics , 1991 .

[11]  Peter Muller,et al.  Alternatives to the Gibbs Sampling Scheme , 1992 .

[12]  Martin Abba Tanner,et al.  Tools for Statistical Inference: Observed Data and Data Augmentation Methods , 1993 .

[13]  D. Spiegelhalter,et al.  Modelling Complexity: Applications of Gibbs Sampling in Medicine , 1993 .

[14]  J. Besag,et al.  Spatial Statistics and Bayesian Computation , 1993 .

[15]  Bradley P. Carlin,et al.  Tools for Statistical Inference: Observed Data and Data Augmentation Methods (Martin A. Tanner) , 1993, SIAM Rev..

[16]  L. Tierney.  Markov Chains for Exploring Posterior Distributions , 1994 .

[17]  D. Berry,et al.  Applications of Lindley Information Measure to the Design of Clinical Experiments , 1994 .

[18]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .