Validation and calibration of probabilistic predictions in ecology

Summary Predictive models in ecology are important for guiding policy and management. However, they are necessarily abstractions of natural systems, making predictive validation imperative. Models, which make predictions about binary outcomes (e.g. Species distribution models, population viability analysis, disease/invasion models), are widespread in the ecological literature. When supporting probability-based management decisions, these predictions need to be assessed with respect to the degree to which predicted probabilities agree with future outcomes. Many predictive models are not validated using external data and are often only assessed in terms of their ability to discriminate between outcomes rather than the degree to which they predicted the correct probabilities. We develop a novel Validation Metric Applied to Probabilistic Predictions (VMAPP), which provides a goodness-of-fit test of calibration for probabilistic prediction models using binary data (e.g. presences and absences in models of species distributions). We analyse the theoretic properties of this test and compare its performance against existing methods, and apply it to a published model in invasion biology, which forecasts the establishment probability of the zooplanktivorous spiny water flea (Bythotrephes longimanus). We selected 102 additional sites to sample four years after the training data were collected and use this independently collected data to assess predictive reliability using VMAPP. Theoretic simulation analysis shows that VMAPP outperforms existing metrics (Cox's regression technique and Hosmer & Lemeshow's χ2 test) in terms of statistical power to identify model miscalibration. Further, we find that under realistic conditions where model parameters are estimated (and have associated uncertainty) that VMAPP is more robust, retaining the appropriate type-I error rates (5%) where previous metrics fail (≤17%). Application of VMAPP to a published invasion model using empirical validation data shows that in addition to having high discriminative power, the model's probabilistic predictions agree with the observed outcomes as measured by VMAPP. We argue that quantifying ecological predictions as probabilities with associated uncertainty provides the most useful information to support management decisions. Ecological predictions, while uncertain, should still be rigorously validated. Identifying the circumstances in which our predictions deviate from observation can further inform the next generation of the model, bringing prediction and reality ever closer.