On the Sample Information About Parameter and Prediction

The Bayesian measure of sample information about the parameter, known as Lindley's measure, is widely used in various problems such as developing prior distributions, models for the likelihood functions and optimal designs. The predictive information is defined similarly and used for model selection and optimal designs, though to a lesser extent. The parameter and predictive information measures are proper utility functions and have been also used in combination. Yet the relationship between the two measures and the effects of conditional dependence between the observable quantities on the Bayesian information measures remain unexplored. We address both issues. The relationship between the two information measures is explored through the information provided by the sample about the parameter and prediction jointly. The role of dependence is explored along with the interplay between the information measures, prior and sampling design. For the conditionally independent sequence of observable quantities, decompositions of the joint information characterize Lindley's measure as the sample information about the parameter and prediction jointly and the predictive information as part of it. For the conditionally dependent case, the joint information about parameter and prediction exceeds Lindley's measure by an amount due to the dependence. More specific results are shown for the normal linear models and a broad subfamily of the exponential family. Conditionally independent samples provide relatively little information for prediction, and the gap between the parameter and predictive information measures grows rapidly with the sample size.

[1]  石川 武司,et al.  On Maximum Likelihood Prediction , 1997 .

[2]  Mohsen Pourahmadi,et al.  Prediction Variance and Information Worth of Observations in Time Series , 2000 .

[3]  D. Berry,et al.  Applications of Lindley Information Measure to the Design of Clinical Experiments , 1994 .

[4]  G. Stewart Collinearity and Least Squares Regression , 1987 .

[5]  Ehsan S. Soofi,et al.  Effects of collinearity on information about regression coefficients , 1990 .

[6]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .

[7]  D. Lindley The Use of Prior Probability Distributions in Statistical Inference and Decisions , 1961 .

[8]  D. Farnsworth A First Course in Order Statistics , 1993 .

[9]  J. Bernardo Reference Analysis , 2005 .

[10]  N. Ebrahimi,et al.  Information Measures in Perspective , 2010 .

[11]  I. Verdinelli,et al.  Bayesian designs for maximizing information and outcome , 1992 .

[12]  E. Soofi Principal component regression under exchangeability , 1988 .

[13]  Seymour Geisser,et al.  8. Predictive Inference: An Introduction , 1995 .

[14]  Nader Ebrahimi,et al.  Ordering univariate distributions by entropy and variance , 1999 .

[15]  Nicholas G. Polson On the Expected Amount of Information from a Non‐Linear Model , 1992 .

[16]  A. Yuan,et al.  An Information Criterion for Likelihood Selection , 1999, IEEE Trans. Inf. Theory.

[17]  On the relative efficiency of grouped and censored survival data , 1989 .

[18]  Nozer D. Singpurwalla,et al.  To survive or to fail: That is the question , 1994 .

[19]  J. Bernardo,et al.  Bayesian Hypothesis Testing: a Reference Approach , 2002 .

[20]  J. Bernardo Expected Information as Expected Utility , 1979 .

[21]  Evgueni A. Haroutunian,et al.  Information Theory and Statistics , 2011, International Encyclopedia of Statistical Science.

[22]  Nader Ebrahimi,et al.  Information properties of order statistics and spacings , 2004, IEEE Transactions on Information Theory.

[23]  G. M. El-Sayyad,et al.  Information and Sampling from the Exponential Distribution , 1969 .

[24]  On the Relative Efficiency of Two Paired‐Data Experiments , 1980 .

[25]  D. Owen Handbook of Mathematical Functions with Formulas , 1965 .

[26]  A. Zellner Optimal Information Processing and Bayes's Theorem , 1988 .

[27]  R. Fildes Conditioning Diagnostics: Collinearity and Weak Data in Regression , 1993 .

[28]  Matthew West,et al.  Bayesian factor regression models in the''large p , 2003 .

[29]  Fulvio Spezzaferri,et al.  A Predictive Model Selection Criterion , 1984 .

[30]  J. Aitchison Goodness of prediction fit , 1975 .

[31]  B. Carlin An Expected Utility Approach to Influence Diagnostics , 1991 .

[32]  K. Chaloner,et al.  Bayesian Experimental Design: A Review , 1995 .

[33]  David Lindley,et al.  BINOMIAL SAMPLING SCHEMES AND THE CONCEPT OF INFORMATION , 1957 .

[34]  Edward I. George,et al.  Admissible predictive density estimation , 2008 .

[35]  Nicholas G. Polson,et al.  Diagnostic Measures for Model Criticism , 1996 .

[36]  D. Lindley On a Measure of the Information Provided by an Experiment , 1956 .

[37]  Fernando A. Quintana,et al.  Nonparametric Bayesian data analysis , 2004 .

[38]  Nader Ebrahimi,et al.  Relative information loss under Type II censored exponential data , 1990 .

[39]  B. Arnold,et al.  A first course in order statistics , 1994 .

[40]  I. Verdinelli,et al.  A note on Bayes designs for inference using a hierarchical linear model , 1980 .

[41]  M. Lawera Predictive inference : an introduction , 1995 .

[42]  David A. Belsley,et al.  Conditioning Diagnostics: Collinearity and Weak Data in Regression , 1991 .

[43]  Xiao-Li Meng,et al.  Quantifying the Fraction of Missing Information for Hypothesis Testing in Statistical and Genetic Studies , 2011, 1102.2774.

[44]  Martin S. Levy,et al.  Goodness of Prediction Fit for Multivariate Linear Models , 1996 .

[45]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[46]  Nader Ebrahimi Prediction intervals for future failures in the exponential distribution under hybrid censoring , 1992 .

[47]  J. Ibrahim,et al.  Power prior distributions for regression models , 2000 .

[48]  Richard E. Barlow,et al.  Expected Information from a Life Test Experiment. , 1983 .

[49]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[50]  J. F. Lawless,et al.  A Prediction Problem Concerning Samples From the Exponential Distribution, With Application in Life Testing , 1971 .

[51]  M. Stone Application of a Measure of Information to the Design and Comparison of Regression Experiments , 1959 .

[52]  On the loss of information through censoring , 1982 .

[53]  I. Verdinelli,et al.  Shannon Information and Bayesian Design for Prediction in Accelerated Life-Testing , 1993 .

[54]  X-L Meng Quantifying the Fraction of Missing Information for Hypothesis Testing in Statistical and Genetic Studies , 2009 .