There are natural scores: Full comment on Shafer,"Testing by betting: A strategy for statistical and scientific communication"

Shafer (2021) offers a betting perspective on statistical testing which may be useful for foundational debates, given that disputes over such testing continue to be intense. To be helpful for researchers, however, this perspective will need more elaboration using real examples in which (a) the betting score has a justification and interpretation in terms of study goals that distinguishes it from the uncountable mathematical possibilities, and (b) the assumptions in the sampling model are uncertain. On justification, Shafer says 'No one has made a convincing case for any particular choice' of a score derived from a P-value and then states that 'the choice is fundamentally arbitrary'. Yet some (but not most) scores can be motivated by study goals (e.g., information measurement; decision making). The one I have seen repeatedly in information statistics and data mining is the surprisal, logworth or S-value s = -log(p), where the log base determines the scale. The present comment explains the rationale for this choice.

[1]  M. J. Bayarri,et al.  Calibration of ρ Values for Testing Precise Null Hypotheses , 2001 .

[2]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[3]  James M. Robins,et al.  Asymptotic Distribution of P Values in Composite Null Models , 2000 .

[4]  Sander Greenland,et al.  Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values , 2019, The American Statistician.

[5]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[6]  Irving John Good,et al.  The Surprise Index for the Multivariate Normal Distribution , 1956 .

[7]  M. J. Bayarri,et al.  P Values for Composite Null Models , 2000 .

[8]  Irving John Good,et al.  Some Logic and History of Hypothesis Testing , 1981 .

[9]  Glenn Shafer,et al.  Author's reply to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer , 2021, Journal of the Royal Statistical Society: Series A (Statistics in Society).

[10]  Leonard A Stefanski,et al.  P-Value Precision and Reproducibility , 2011, The American statistician.

[11]  N. Lazar,et al.  Moving to a World Beyond “p < 0.05” , 2019, The American Statistician.

[12]  S. Greenland,et al.  Inferential statistics are descriptive statistics , 2018 .

[13]  S. Greenland,et al.  Semantic and Cognitive Tools to Aid Statistical Inference: Replace Confidence and Significance by Compatibility and Surprise , 2019, 1909.08579.