Simple and statistically sound recommendations for analysing physical theories

Physical theories that depend on many parameters or are tested against data from many different experiments pose unique challenges to statistical inference. Many models in particle physics, astrophysics and cosmology fall into one or both of these categories. These issues are often sidestepped with statistically unsound ad hoc methods, involving intersection of parameter intervals estimated by multiple experiments, and random or grid sampling of model parameters. Whilst these methods are easy to apply, they exhibit pathologies even in low-dimensional parameter spaces, and quickly become problematic to use and interpret in higher dimensions. In this article we give clear guidance for going beyond these procedures, suggesting where possible simple methods for performing statistically sound inference, and recommendations of readily-available software tools and standards that can assist in doing so. Our aim is to provide any physicists lacking comprehensive statistical training with recommendations for reaching correct scientific conclusions, with only a modest increase in analysis burden. Our examples can be reproduced with the code publicly available at Zenodo.

[1]  D. Lakens The Practical Alternative to the p Value Is the Correctly Used p Value , 2021, Perspectives on psychological science : a journal of the Association for Psychological Science.

[2]  Eduardo C. Garrido-Merch'an,et al.  A comparison of optimisation algorithms for high-dimensional particle and astrophysics applications , 2021, Journal of High Energy Physics.

[3]  Gonzalo,et al.  scikit-hep/iminuit: v1.5.4 , 2020 .

[4]  K. Cranmer,et al.  Simulation-Based Inference Methods for Particle Physics , 2020, Artificial Intelligence for High Energy Physics.

[5]  Robert D. Cousins,et al.  What is the likelihood function, and how is it used in particle physics? , 2020, 2010.00356.

[6]  L. Lyons,et al.  Reproducibility and Replication of Experimental Particle Physics Results , 2020, Issue 2.4, Fall 2020.

[7]  R. B. Barreiro,et al.  Planck 2018 results: V. CMB power spectra and likelihoods , 2020 .

[8]  Christian P. Robert,et al.  Computing Bayes: Bayesian Computation from 1763 to the 21st Century , 2020, 2004.06425.

[9]  David W. Miller,et al.  Reinterpretation of LHC Results for New Physics: Status and recommendations after Run 2 , 2020, SciPost Physics.

[10]  Jihyun Bhom,et al.  HEPLike: An open source framework for experimental likelihood evaluation , 2020, Comput. Phys. Commun..

[11]  Joel Nothman,et al.  Author Correction: SciPy 1.0: fundamental algorithms for scientific computing in Python , 2020, Nature Methods.

[12]  Avrim Blum,et al.  Foundations of Data Science , 2020 .

[13]  J. Aalbers,et al.  Searching for new phenomena with profile likelihood ratio tests , 2019, Nature Reviews Physics.

[14]  M. Pierini,et al.  HEPfit: a code for the combination of indirect and direct constraints on high energy physics models , 2019, The European Physical Journal C.

[15]  T. Kuhr,et al.  Averages of b-hadron, c-hadron, and $$\tau $$-lepton properties as of 2018 , 2019, The European Physical Journal C.

[16]  Johannes L. Schönberger,et al.  SciPy 1.0: fundamental algorithms for scientific computing in Python , 2019, Nature Methods.

[17]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[18]  Pietro Vischia,et al.  Reporting results in High Energy Physics publications: A manifesto , 2019, Reviews in Physics.

[19]  J. Speagle dynesty: a dynamic nested sampling package for estimating Bayesian posteriors and evidences , 2019, Monthly Notices of the Royal Astronomical Society.

[20]  R. Cousins,et al.  Lectures on Statistics in Theory: Prelude to Statistics in Practice. , 2018, 1807.05996.

[21]  Peter Athron,et al.  Impact of vacuum stability, perturbativity and XENON1T on global fits of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document , 2018, The European Physical Journal C.

[22]  J. Lesgourgues,et al.  MontePython 3: Boosted MCMC sampler and other features , 2018, Physics of the Dark Universe.

[23]  Nicholas P. Holmes,et al.  Justify your alpha , 2018, Nature Human Behaviour.

[24]  Jonathan M. Cornell,et al.  GAMBIT: the global and modular beyond-the-standard-model inference tool , 2018, The European Physical Journal C.

[25]  Jonathan M. Cornell,et al.  GAMBIT: the global and modular beyond-the-standard-model inference tool , 2017, The European Physical Journal C.

[26]  Daniel Foreman-Mackey,et al.  Data Analysis Recipes: Using Markov Chain Monte Carlo , 2017, 1710.06068.

[27]  David Gal,et al.  Abandon Statistical Significance , 2017, The American Statistician.

[28]  Coryn A. L. Bailer-Jones,et al.  Practical Bayesian Inference: A Primer for Physical Scientists , 2017 .

[29]  J. Conrad,et al.  Comparison of statistical sampling methods with ScannerBit, the GAMBIT scanning module , 2017, 1705.07959.

[30]  C. Rogan,et al.  ColliderBit: a GAMBIT module for the calculation of high-energy collider observables and likelihoods , 2017, The European Physical Journal C.

[31]  Jonathan M. Cornell,et al.  Global fits of GUT-scale SUSY models with GAMBIT , 2017, The European Physical Journal C.

[32]  S. Goodman,et al.  Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations , 2016, European Journal of Epidemiology.

[33]  Qiang Yuan,et al.  LikeDM: Likelihood calculator of dark matter detection , 2016, Comput. Phys. Commun..

[34]  P. O. Hulth,et al.  Improved limits on dark matter annihilation in the Sun with the 79-string IceCube detector and implications for supersymmetry , 2016, 1601.00653.

[35]  Jeffrey N. Rouder,et al.  The fallacy of placing confidence in confidence intervals , 2015, Psychonomic bulletin & review.

[36]  L. Rauch,et al.  Dark matter direct-detection experiments , 2015, 1509.08767.

[37]  M. P. Hobson,et al.  polychord: nested sampling for cosmology , 2015, Monthly Notices of the Royal Astronomical Society: Letters.

[38]  M. Baak,et al.  The global electroweak fit at NNLO and prospects for the LHC and ILC , 2014, The European Physical Journal C.

[39]  Wolfgang von der Linden,et al.  Bayesian Probability Theory: Applications in the Physical Sciences , 2014 .

[40]  M. Baak,et al.  The global electroweak fit at NNLO and prospects for the LHC and ILC , 2014, 1407.3792.

[41]  Paul A Murtaugh,et al.  In defense of P values. , 2014, Ecology.

[42]  Robert D. Cousins,et al.  The Jeffreys–Lindley paradox and discovery criteria in high energy physics , 2013, Synthese.

[43]  Louis Lyons,et al.  Discovering the Significance of 5 sigma , 2013, 1310.1284.

[44]  Gregory Schott,et al.  Data Analysis in High Energy Physics: A Practical Guide to Statistical Methods , 2013 .

[45]  K. Cranmer,et al.  Erratum to: Asymptotic formulae for likelihood-based tests of new physics , 2013 .

[46]  The Cms Collaboration Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC , 2012, 1207.7235.

[47]  P. O. Hulth,et al.  Use of event-level neutrino telescope data in global fits for theories of new physics , 2012, 1207.0810.

[48]  L. Roszkowski,et al.  The CMSSM Favoring New Territories: The Impact of New LHC Limits and a 125 GeV Higgs , 2012 .

[49]  L. Roszkowski,et al.  Constrained MSSM favoring new territories: the impact of new LHC limits and a 125 GeV Higgs boson , 2012, 1206.0264.

[50]  P. Bechtle,et al.  Constrained supersymmetry after two years of LHC data: a global view with Fittino , 2012, 1204.4199.

[51]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[52]  Daniel Foreman-Mackey,et al.  emcee: The MCMC Hammer , 2012, 1202.3665.

[53]  Vinesh Rajpaul,et al.  Genetic algorithms in astronomy and astrophysics , 2012, ArXiv.

[54]  R. Trotta,et al.  Fundamental statistical limitations of future dark matter direct detection experiments , 2012, 1201.3631.

[55]  A. Roeck,et al.  Higgs and supersymmetry , 2011, 1112.3564.

[56]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[57]  F. Feroz,et al.  A coverage study of the CMSSM based on ATLAS sensitivity using fast neural networks techniques , 2010, 1011.4306.

[58]  P. Scott,et al.  Statistical coverage for supersymmetric parameter estimation : a case study with direct detection of dark matter , 2010, 1011.4297.

[59]  K. Cranmer,et al.  Asymptotic formulae for likelihood-based tests of new physics , 2010, 1007.1727.

[60]  E. Gross,et al.  Trial factors for the look elsewhere effect in high energy physics , 2010, 1005.1891.

[61]  R. Cousins,et al.  Frequentist evaluation of intervals estimated for a binomial parameter and for the ratio of Poisson means , 2009, 0905.3831.

[62]  F. Feroz,et al.  MultiNest: an efficient and robust Bayesian inference tool for cosmology and particle physics , 2008, 0809.3437.

[63]  George Casella,et al.  A Short History of Markov Chain Monte Carlo: Subjective Recollections from Incomplete Data , 2008, 0808.2902.

[64]  R. Cousins Comment on "Bayesian analysis of pentaquark signals from CLAS data". , 2008, Physical review letters.

[65]  S. Goodman A dirty dozen: twelve p-value misconceptions. , 2008, Seminars in hematology.

[66]  R. Trotta Bayes in the sky: Bayesian inference and model selection in cosmology , 2008, 0803.4089.

[67]  R. Hubbard,et al.  Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing , 2008 .

[68]  E. Wagenmakers A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[69]  C. Lester,et al.  Natural priors, CMSSM fits and LHC weather forecasts , 2007, 0705.0487.

[70]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[71]  J. Skilling Nested sampling for general Bayesian computation , 2006 .

[72]  Tony O’Hagan Bayes factors , 2006 .

[73]  R. Trotta,et al.  A Markov chain Monte Carlo analysis of the CMSSM , 2006, hep-ph/0602028.

[74]  G. Punzi Ordering algorithms and confidence intervals in the presence of nuisance parameters , 2005, physics/0511202.

[75]  George Woodworth,et al.  Bayesian Reasoning in Data Analysis: A Critical Introduction , 2004 .

[76]  Wolfgang A. Rolke,et al.  Limits and confidence intervals in the presence of nuisance parameters , 2004, physics/0403059.

[77]  M. J. Bayarri,et al.  Confusion Over Measures of Evidence (p's) Versus Errors (α's) in Classical Statistical Testing , 2003 .

[78]  A. Read Presentation of search results: the CLs technique , 2002 .

[79]  Bertolt Brecht The manifesto , 2002 .

[80]  M. Ciuchini,et al.  2000 CKM-triangle analysis a critical review with updated experimental inputs and theoretical parameters , 2000, hep-ph/0012308.

[81]  W. Jason Owen,et al.  Statistical Data Analysis , 2000, Technometrics.

[82]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[83]  R. Cousins,et al.  A Unified Approach to the Classical Statistical Analysis of Small Signals , 1997, physics/9711021.

[84]  L. Wasserman,et al.  The Selection of Prior Distributions by Formal Rules , 1996 .

[85]  M. Schervish P Values: What They are and What They are Not , 1996 .

[86]  Alan D. Martin,et al.  Review of Particle Physics , 2018, Physical Review D.

[87]  P. Charbonneau Genetic algorithms in astronomy and astrophysics , 1995 .

[88]  E. Greenfield Critical pathways: what they are and what they are not. , 1995, The Journal of burn care & rehabilitation.

[89]  A. W. Kemp,et al.  Kendall's Advanced Theory of Statistics. , 1994 .

[90]  J. Berger,et al.  Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence , 1987 .

[91]  Donald B. Rubin,et al.  Efficiently Simulating the Coverage Properties of Interval Estimates , 1986 .

[92]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[93]  F. James Statistical Methods in Experimental Physics , 1973 .

[94]  J. Dickey The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters , 1971 .

[95]  Edwin T. Jaynes,et al.  Prior Probabilities , 1968, Encyclopedia of Machine Learning.

[96]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[97]  D. Lindley A STATISTICAL PARADOX , 1957 .

[98]  S. S. Wilks The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses , 1938 .

[99]  J. Neyman Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability , 1937 .

[100]  Simplified likelihood for the re-interpretation of public CMS results , 2017 .

[101]  Christopher D. Chambers,et al.  Redefine statistical significance , 2017, Nature Human Behaviour.

[102]  Tanja Hueber,et al.  Statistics For Nuclear And Particle Physicists , 2016 .

[103]  Doreen Eichel,et al.  Data Analysis A Bayesian Tutorial , 2016 .

[104]  Matan Gavish,et al.  THE IRRECONCILABILITY OF P-VALUES AND EVIDENCE , 2010 .

[105]  Phil Gregory,et al.  Bayesian Logical Data Analysis for the Physical Sciences: Probability theory as extended logic , 2005 .

[106]  Dieter Gstach,et al.  Markov Chain Monte Carlo Analysis , 2002 .

[107]  S. Senn Two cheers for P-values? , 2001, Journal of epidemiology and biostatistics.

[108]  James O. Berger,et al.  Objective Bayesian Methods for Model Selection: Introduction and Comparison , 2001 .

[109]  R D Cousins,et al.  Comments on methods for setting confidence limits , 2000 .

[110]  Robert Cedergren,et al.  Guided tour , 1990, Nature.

[111]  Matts Roos,et al.  MINUIT-a system for function minimization and analysis of the parameter errors and correlations , 1984 .

[112]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[113]  H. H. Rosenbrock,et al.  An Automatic Method for Finding the Greatest or Least Value of a Function , 1960, Comput. J..

[114]  M. Kendall Statistical Methods for Research Workers , 1937, Nature.

[115]  E. S. Pearson,et al.  On the Problem of the Most Efficient Tests of Statistical Hypotheses , 1933 .

[116]  L. M. M.-T. Theory of Probability , 1929, Nature.