A General Analytic Method for Assessing Sensitivity to Bias of Performance Measures for Dichotomous Forecasts

Abstract Performance measures computed from the 2 × 2 contingency table of outcomes for dichotomous forecasts are sensitive to bias. The method presented here evaluates how the probability of detection (POD) must change as bias changes so that a performance measure improves at a given value of bias. A critical performance ratio (CPR) of the change of POD to the change in bias is derived for a number of performance measures. If a change in POD associated with a bias change satisfies the CPR condition, the performance measure will indicate an improved forecast. If a perfect measure of performance existed, it would always exhibit its optimal value at bias equal to one. Actual measures of performance are susceptible to bias, indicating a better forecast for bias values not equal to one. The CPR is specifically applied to assess the conditions for an improvement toward a more favorable value of several commonly used performance measures as bias increases or decreases through the value one. All performance meas...