Calibration of ρ Values for Testing Precise Null Hypotheses

P values are the most commonly used tool to measure evidence against a hypothesis or hypothesized model. Unfortunately, they are often incorrectly viewed as an error probability for rejection of the hypothesis or, even worse, as the posterior probability that the hypothesis is true. The fact that these interpretations can be completely misleading when testing precise hypotheses is first reviewed, through consideration of two revealing simulations. Then two calibrations of a ρ value are developed, the first being interpretable as odds and the second as either a (conditional) frequentist error probability or as the posterior probability of the hypothesis.

[1]  H. Jeffreys,et al.  Theory of probability , 1896 .

[2]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[3]  Taylor Francis Online,et al.  The American statistician , 1947 .

[4]  A. Birnbaum.  ON THE FOUNDATIONS OF STATISTICAL INFERENCE: BINARY EXPERIMENTS' , 1961 .

[5]  Ward Edwards,et al.  Bayesian Statistical Inference for Psychological Research , 1963 .

[6]  Jean D. Gibbons,et al.  P-values: Interpretation and Methodology , 1975 .

[7]  J. Kiefer.  Conditional Confidence Statements and Confidence Estimators , 1977 .

[8]  James O. Berger,et al.  Statistical Decision Theory and Bayesian Analysis, Second Edition , 1985 .

[9]  J. Berger,et al.  Testing Precise Hypotheses , 1987 .

[10]  J. Berger,et al.  Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence , 1987 .

[11]  G. Casella,et al.  Reconciling Bayesian and Frequentist Evidence in the One-Sided Testing Problem , 1987 .

[12]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[13]  James O. Berger,et al.  Lower bounds on Bayes factors for multinomial distributions, with application to chi-squared tests of fit , 1990 .

[14]  J. Q. Smith,et al.  1. Bayesian Statistics 4 , 1993 .

[15]  James O. Berger,et al.  An overview of robust Bayesian analysis , 1994 .

[16]  L. Brown,et al.  A Unified Conditional Frequentist and Bayesian Test for Fixed and Sequential Simple Hypothesis Testing , 1994 .

[17]  A. O'Hagan,et al.  Fractional Bayes factors for model comparison , 1995 .

[18]  J. Berger,et al.  The Intrinsic Bayes Factor for Model Selection and Prediction , 1996 .

[19]  Robert L. Wolpert,et al.  Testing Simple Hypotheses , 1996 .

[20]  M. Schervish.  P Values: What They are and What They are Not , 1996 .

[21]  L. Wasserman,et al.  Computing Bayes Factors by Combining Simulation and Asymptotic Approximations , 1997 .

[22]  T. Louis,et al.  Unified frequentist and Bayesian testing of a precise hypothesis , 1997 .

[23]  Unified Bayesian and conditional frequentist testing procedures , 1998 .

[24]  S. Goodman.  Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy , 1999, Annals of Internal Medicine.

[25]  Rafe M. J. Donahue,et al.  A Note on Information Seldom Reported via the P Value , 1999 .

[26]  E. Samuel-Cahn,et al.  P Values as Random Variables—Expected P Values , 1999 .

[27]  Steven Goodman.  Toward Evidence-Based Medical Statistics. 2: The Bayes Factor , 1999, Annals of Internal Medicine.

[28]  Julia Mortera,et al.  Default Bayes Factors for Nonnested Hypothesis Testing , 1999 .

[29]  S. Goodman,et al.  Toward Evidence-Based Medical Statistics. 2: The Bayes Factor , 1999, Annals of Internal Medicine.

[30]  M. J. Bayarri,et al.  P Values for Composite Null Models , 2000 .

[31]  Luis R. Pericchi,et al.  ACCURATE AND STABLE BAYESIAN MODEL SELECTION: THE MEDIAN INTRINSIC BAYES FACTOR* , 2002 .