Frequentist and Bayesian confidence intervals

Frequentist (classical) and Bayesian approaches to the construction of confidence limits are compared. Various examples which illustrate specific problems are presented. The Likelihood Principle and the Stopping Rule Paradox are discussed. The performance of the different methods is investigated relative to the properties coherence, precision, bias, universality, simplicity. A proposal on how to define error limits in various cases are derived from the comparison. They are based on the likelihood function only and follow in most cases the general practice in high energy physics. Classical methods are not recommended because they violate the Likelihood Principle, they can produce inconsistent results, suffer from lack of precision and generality. Also the extreme Bayesian approach with arbitrary choice of the prior probability density or priors deduced from scaling laws is rejected.

[1]  Gunter Zech,et al.  Upper limits in experiments with background or measurement errors , 1989 .

[2]  Byron P. Roe,et al.  Probability and Statistics in Experimental Physics , 1992 .

[3]  G. D'Agostini.  Bayesian reasoning versus conventional statistics in High Energy Physics , 1998 .

[4]  G. C. Tiao,et al.  Bayesian inference in statistical analysis , 1973 .

[5]  J. Neyman.  Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability , 1937 .

[6]  Ward Edwards,et al.  Bayesian Statistical Inference for Psychological Research , 1963 .

[7]  Ronald Aylmer Sir Fisher,et al.  Statistical Methods, Experimental Design, and Scientific Inference , 1990 .

[8]  Williams.  Comment on "Small-signal analysis in high-energy physics: A Bayesian approach" , 1988, Physical review. D, Particles and fields.

[9]  Limits on the Majorana neutrino mass in the 0.1-eV range , 1999, hep-ex/9902014.

[10]  Small signal with background: objective confidence intervals and regions for physical parameters from the principle of maximum likelihood , 1998 .

[11]  G. Zech.  Comparing statistical data to Monte Carlo simulation: Parameter fitting and unfolding , 1995 .

[12]  Leonard J. Savage,et al.  The Foundations of Statistics Reconsidered , 1961 .

[13]  H. Leon Harter,et al.  Criteria for Best Substitute Interval Estimators, with an Application to the Normal Distribution , 1964 .

[14]  E. Iso,et al.  Measurement Uncertainty and Probability: Guide to the Expression of Uncertainty in Measurement , 1995 .

[15]  R D Cousins,et al.  Comments on methods for setting confidence limits , 2000 .

[16]  Rory A. Fisher,et al.  Theory of Statistical Estimation , 1925, Mathematical Proceedings of the Cambridge Philosophical Society.

[17]  James O. Berger,et al.  Statistical Analysis and the Illusion of Objectivity , 1988 .

[18]  C. Giunti.  New ordering principle for the classical statistical analysis of Poisson processes with background , 1999 .

[19]  J. Kiefer.  Conditional Confidence Statements and Confidence Estimators , 1977 .

[20]  R. Cousins.  Why isn’t every physicist a Bayesian? , 1995 .

[21]  Confronting classical and Bayesian confidence limits to examples , 2000, hep-ex/0004011.

[22]  E. al.,et al.  Incorporation of the statistical uncertainty in the background estimate into the upper limit on the signal , 1997, physics/9707020.

[23]  G. D'Agostini,et al.  Confidence limits: what is the problem? Is there the solution? , 2000, hep-ex/0002055.

[24]  D. Freedman,et al.  On inconsistent Bayes estimates of location , 1986 .

[25]  Harold Jeffreys,et al.  Further significance tests , 1936, Mathematical Proceedings of the Cambridge Philosophical Society.

[26]  T. A. Bancroft,et al.  Statistical theory and inference in research , 1982 .

[27]  James O. Berger,et al.  Ockham's Razor and Bayesian Analysis , 1992 .

[28]  Charles Stein,et al.  A Remark on the Likelihood Principle , 1962 .

[29]  Estimation of upper limits using a Poisson statistic , 1999, hep-ex/9904025.

[30]  K. Pearson,et al.  Biometrika , 1902, The American Naturalist.

[31]  P. Lunde,et al.  General background. , 1993, WHO regional publications. European series.

[32]  D. Basu.  17 – Relevance of Randomization in Data Analysis , 1978 .

[33]  Jeffreys priors versus experienced physicist priors - arguments against objective Bayesian theory , 1998, physics/9811045.

[34]  Mark Mandelkern,et al.  The statistical analysis of Gaussian and Poisson signals near physical boundaries , 2000 .

[35]  John D. Kalbfleisch,et al.  Application of Likelihood Methods to Models Involving Large Numbers of Parameters , 1970 .

[36]  A. Read.  MODIFIED FREQUENTIST ANALYSIS OF SEARCH RESULTS ( THE , 2000 .

[37]  Irving John Good,et al.  Some Logic and History of Hypothesis Testing , 1981 .

[38]  Allan Birnbaum,et al.  More on Concepts of Statistical Evidence , 1972 .

[39]  T. Junk.  Confidence Level Computation for Combining Searches with Small Statistics , 1999, hep-ex/9902006.

[40]  B. Efron.  Why Isn't Everyone a Bayesian? , 1986 .

[41]  G. Punzi.  A stronger classical definition of Confidence Limits , 1999, hep-ex/9912048.

[42]  Bruce M. Hill,et al.  On some statistical paradoxes and non-conglomerability , 1980 .

[43]  Prosper.  Small-signal analysis in high-energy physics: A Bayesian approach. , 1988, Physical review. D, Particles and fields.

[44]  R. Cousins,et al.  A Unified Approach to the Classical Statistical Analysis of Small Signals , 1997, physics/9711021.

[45]  E. Lehmann,et al.  Testing Statistical Hypothesis. , 1960 .

[46]  A. Read.  Modified frequentist analysis of search results (The CL(s) method) , 2000 .

[47]  Gunter Zech.  Objections to the Unified Approach to the Computation of Classical Confidence Limits , 1998 .

[48]  W. D. Ray.  The Foundation of Statistical Inference , 1963 .

[49]  J. K. Ghosh.  Statistical Information and Likelihood , 1988 .

[50]  W. R. Buckland,et al.  A dictionary of statistical terms , 1958 .

[51]  Byron P. Roe,et al.  Improved probability method for estimating signal in the presence of background , 1998, physics/9812036.

[52]  Mervyn Stone,et al.  Strong Inconsistency from Uniform Priors , 1976 .

[53]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[54]  V. P. Godambe,et al.  Memorial Article: Allan Birnbaum 1923-1976 , 1982 .

[55]  Hayes,et al.  Review of Particle Physics. , 1996, Physical review. D, Particles and fields.

[56]  C. Stein,et al.  Estimation with Quadratic Loss , 1992 .

[57]  G. Barnard,et al.  Likelihood Inference and Time Series , 1962 .

[58]  R. Fisher.  The Advanced Theory of Statistics , 1943, Nature.

[59]  G. A. Barnard,et al.  THE LOGIC OF STATISTICAL INFERENCE1 , 1972, The British Journal for the Philosophy of Science.

[60]  G. W. Snedecor.  Statistical Methods , 1964 .