Assessing the performance of real-time epidemic forecasts: A case study of the 2013-16 Ebola epidemic

Real-time forecasts based on mathematical models can inform critical decision-making during infectious disease outbreaks. Yet, epidemic forecasts are rarely evaluated during or after the event, and there is little guidance on what the best metrics for assessment are. Here, we propose to disentangle different components of forecasting ability by using metrics that separately assess the calibration, sharpness and unbiasedness of forecasts. We used this approach to analyse the performance of weekly forecasts generated in real time in Western Area, Sierra Leone, during the 2013–16 Ebola epidemic in West Africa. We found that probabilistic calibration was good at short time horizons but deteriorated for long-term forecasts. This suggests that forecasts provided usable performance only a few weeks ahead of time, reflecting the high level of uncertainty in the processes driving the trajectory of the epidemic. Comparing the semi-mechanistic model we used during the epidemic to simpler null models showed that the our model performed better with respect to probabilistic calibration, and that this would have been identified from the earliest stages of the outbreak. As forecasts become a routine part of the toolkit in public health, standards for evaluation of performance will be important for assessing quality and improving credibility of mathematical models, and for elucidating difficulties and trade-o s when aiming to make the most useful and reliable forecasts.