Evaluation of Two Methods for Incorporating a Systematic Uncertainty into a Test of the Background-only Hypothesis for a Poisson Process

Hypothesis tests for the presence of new sources of Poisson counts amidst background processes are frequently performed in high energy physics, gamma ray astronomy, and other branches of science. This talk briefly summarizes work in which we evaluate two classes of algorithms for dealing with uncertainty in the mean background in such tests. This talk briefly summarizes studies, performed with Robert Cousins and described in Ref. [1], of two methods for incorporating a systematic uncertainty into a test of the background-only hypothesis for a Poisson process. In a situation common in both gamma-ray astronomy (GRA) and high-energy physics (HEP), non events are observed from a Poisson process with mean μs + μb; the signal mean μs is of interest, while the background mean μb is a nuisance parameter. In this work, we study tests of the background-only null hypothesis (μs = 0) in two prototypical problems in GRA and HEP as follows. The “on/off” problem. In GRA, non photons are detected with a telescope pointed on-source, i.e. with some putative source in the field of view; and noff photons are detected with the telescope pointed off-source. The ratio τ of observing time toff/ton is assumed known exactly. In the analogous example from HEP, one counts non events in a signal region where one is looking for an excess above background. One observes noff events in a background control (sideband) region where no excess is expected. The ratio τ of sideband to signal region events under the background-only null hypothesis is again assumed known. The “Gaussian-mean background” problem. In another scenario, there is a subsidiary measurement which determines μb with normal (Gaussian) uncertainty with rms deviation σb. We assume σb to be precisely known, either absolutely, or as a set fraction of μb. In either problem, for a data set one can then proceed to calculate the tail probability (p-value) under the null hypothesis. In HEP, one typically quotes the significance S (known in the statistics literature as the Z-value) of the data set, namely the p-value converted to equivalent normal standard deviations. As detailed by Linnemann [2] at PhyStat 2003, there is an approximate correspondence between the two problems. For the on/off problem, an estimate of the mean background in the signal region is