How the Maximal Evidence of P-Values Against Point Null Hypotheses Depends on Sample Size

ABSTRACT Minimum Bayes factors are commonly used to transform two-sided p-values to lower bounds on the posterior probability of the null hypothesis. Several proposals exist in the literature, but none of them depends on the sample size. However, the evidence of a p-value against a point null hypothesis is known to depend on the sample size. In this article, we consider p-values in the linear model and propose new minimum Bayes factors that depend on sample size and converge to existing bounds as the sample size goes to infinity. It turns out that the maximal evidence of an exact two-sided p-value increases with decreasing sample size. The effect of adjusting minimum Bayes factors for sample size is shown in two applications.

[1]  Leonhard Held,et al.  Approximate Bayesian Model Selection with the Deviance Statistic , 2013, 1308.6780.

[2]  L. Held.  Reverse-Bayes analysis of two common misinterpretations of significance tests , 2013, Clinical trials.

[3]  S. Goodman,et al.  Toward Evidence-Based Medical Statistics. 2: The Bayes Factor , 1999, Annals of Internal Medicine.

[4]  V. Johnson.  Revised standards for statistical evidence , 2013, Proceedings of the National Academy of Sciences.

[5]  M. A. Best.  Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .

[6]  S. Goodman.  Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy , 1999, Annals of Internal Medicine.

[7]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[8]  Valen E. Johnson,et al.  Properties of Bayes Factors Based on Test Statistics , 2008 .

[9]  P. Freeman,et al.  The role of p-values in analysing trial results. , 1993, Statistics in medicine.

[10]  S. Goodman,et al.  Of P-values and Bayes: a modest proposal. , 2001, Epidemiology.

[11]  Sander Greenland,et al.  Null misinterpretation in statistical testing and its impact on health risk assessment. , 2011, Preventive medicine.

[12]  M. Clyde,et al.  Mixtures of g Priors for Bayesian Variable Selection , 2008 .

[13]  R. Matthews,et al.  Methods for Assessing the Credibility of Clinical Trial Outcomes , 2001 .

[14]  V. Vovk.  A logic of probability, with application to the foundations of statistics , 1993 .

[15]  Gudmund R. Iversen,et al.  Bayesian statistical inference , 1984 .

[16]  Sander Greenland,et al.  Living with P Values: Resurrecting a Bayesian Perspective on Frequentist Statistics , 2013, Epidemiology.

[17]  Ward Edwards,et al.  Bayesian statistical inference for psychological research. , 1963 .

[18]  E. Wagenmakers.  A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[19]  Jianhua Hu,et al.  Bayesian model selection using test statistics , 2009, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[20]  J. Berger,et al.  Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence , 1987 .

[21]  S. Goodman,et al.  A comment on replication, p-values and evidence. , 1992, Statistics in medicine.

[22]  David J. Spiegelhalter,et al.  An Overview of the Bayesian Approach , 2004 .

[23]  L. Wasserman,et al.  A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion , 1995 .

[24]  Leonhard Held,et al.  Hyper-$g$ priors for generalized linear models , 2010, 1008.1550.

[25]  Sander Greenland,et al.  Bayesian perspectives for epidemiological research: I. Foundations and basic methods. , 2006, International journal of epidemiology.

[26]  R. Royall.  The Effect of Sample Size on the Meaning of Significance Tests , 1986 .

[27]  Jie W Weiss,et al.  Bayesian Statistical Inference for Psychological Research , 2008 .

[28]  J. Copas.  Regression, Prediction and Shrinkage , 1983 .

[29]  James O. Berger,et al.  Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses , 2015, Journal of mathematical psychology.

[30]  M. J. Bayarri,et al.  Calibration of ρ Values for Testing Precise Null Hypotheses , 2001 .

[31]  Valen E. Johnson,et al.  Bayes factors based on test statistics , 2005 .

[32]  L. Held.  A nomogram for P values , 2010, BMC medical research methodology.