Promoting the use of probabilistic weather forecasts through a dialogue between scientists, developers and end‐users

Funding information Hans Ertel Centre for Weather Research, a network of universities, research institutes and the Deutscher Wetterdienst, and funded by the BMVI (Federal Ministry of Transport and Digital Infrastructures). Today’s ensemble weather prediction systems provide reliable and sharp probabilistic forecasts—yet they are still rarely communicated to outside users because of two main worries: the difficulty of communicating probabilities to lay audiences and their presumed reluctance to use probabilistic forecasts. To bridge the gap between the forecasts available and their use in day-to-day decision making, we encourage scientists, developers, and end-users to engage in interdisciplinary collaborations. Here, we discuss our experience with three different approaches of introducing probabilistic forecasts to different user groups and the theoretical and practical challenges that emerged. The approaches range from quantitative analyses of users’ revealed preferences online to a participatory developer–user dialogue based on trial cases and interactive demonstration tools. The examples illustrate three key points. First, to make informed decisions, users need access to probabilistic forecasts. Second, forecast uncertainty can be understood if its visual representations follow validated best practices from risk communication and information design; we highlight five important recommendations from that literature for communicating probabilistic forecasts. Third, to appreciate the value of probabilistic forecasts for their decisions, users need the opportunity to experience them in their everyday practice. With these insights and practical pointers, we hope to support future efforts to integrate probabilistic forecasts into everyday decision making.

[1]  Andrea L. Taylor,et al.  Communicating high impact weather: Improving warnings and decision making processes , 2018, International Journal of Disaster Risk Reduction.

[2]  S. Joslyn,et al.  Decisions With Uncertainty: The Glass Half Full , 2013 .

[3]  Dean P. Foster,et al.  Graininess of judgment under uncertainty: An accuracy-informativeness trade-off , 1995 .

[4]  S. Joslyn,et al.  The Advantages of Predictive Interval Forecasts for Non‐Expert Users and the Impact of Visualizations , 2013 .

[5]  P. Slovic Perception of risk. , 1987, Science.

[6]  Jeffrey K. Lazo,et al.  The Costs and Losses of Integrating Social Sciences and Meteorology , 2010 .

[7]  Limor Nadav-Greenberg,et al.  The effects of wording on the understanding and use of uncertainty information in a threshold forecasting decision , 2009 .

[8]  P. Tetlock Thinking the unthinkable: sacred values and taboo cognitions , 2003, Trends in Cognitive Sciences.

[9]  Elisabeth Stephens,et al.  Communicating probabilistic information from climate model ensembles—lessons from numerical weather prediction , 2012 .

[10]  I. Lipkus Numeric, Verbal, and Visual Formats of Conveying Health Risks: Suggested Best Practices and Future Recommendations , 2007, Medical decision making : an international journal of the Society for Medical Decision Making.

[11]  B. Fischhoff,et al.  Communicating Risks and Benefits: An Evidence Based User's Guide , 2012 .

[12]  P. L. Houtekamer,et al.  A System Simulation Approach to Ensemble Prediction , 1996 .

[13]  Michelle McDowell,et al.  Meta-Analysis of the Effect of Natural Frequencies on Bayesian Reasoning , 2017, Psychological bulletin.

[14]  E. Weber,et al.  Predicting Risk-Sensitivity in Humans and Lower Animals: Risk as Variance or Coefficient of Variation , 2004, Psychological review.

[15]  Gerd Gigerenzer,et al.  Better Doctors, Better Patients, Better Decisions: Envisioning Health Care 2020 , 2011 .

[16]  Daniel G. Goldstein,et al.  Improving Comprehension of Numbers in the News , 2016, CHI.

[17]  Gary E. Bolton,et al.  A Laboratory Study of the Benefits of Including Uncertainty Information in Weather Forecasts , 2006 .

[18]  Anthony Leiserowitz,et al.  Misinterpretations of the “Cone of Uncertainty” in Florida during the 2004 Hurricane Season , 2007 .

[19]  A. H. Murphy,et al.  Misinterpretations of precipitation probability forecasts , 1980 .

[20]  Mike Pearson,et al.  Visualizing Uncertainty About the Future , 2022 .

[21]  Thomas Kox,et al.  Anticipation and Response: Emergency Services in Severe Weather Situations in Germany , 2018, International Journal of Disaster Risk Science.

[22]  Thomas Kox,et al.  Towards user-orientated weather warnings , 2018, International Journal of Disaster Risk Reduction.

[23]  Vivianne H M Visschers,et al.  Probability Information in Risk Communication: A Review of the Research Literature , 2009, Risk analysis : an official publication of the Society for Risk Analysis.

[24]  Bang Wong,et al.  Points of view: Color blindness , 2011, Nature Methods.

[25]  T. Palmer The ECMWF ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years , 2018, Quarterly Journal of the Royal Meteorological Society.

[26]  Gerd Gigerenzer,et al.  “A 30% Chance of Rain Tomorrow”: How Does the Public Understand Probabilistic Weather Forecasts? , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[27]  Christa Fouché,et al.  An Invitation to Dialogue , 2011 .

[28]  Leonard A. Smith,et al.  Combining dynamical and statistical ensembles , 2003 .

[29]  Robin M. Hogarth,et al.  Communicating forecasts: The simplicity of simulated experience , 2015 .

[30]  M A Matos,et al.  Setting the Operating Reserve Using Probabilistic Wind Power Forecasts , 2011, IEEE Transactions on Power Systems.

[31]  Zied Ben Bouallègue,et al.  Statistical postprocessing of ensemble global radiation forecasts with penalized quantile regression , 2017 .

[32]  Jessica S. Ancker,et al.  The Practice of Informatics: Design Features of Graphs in Health Risk Communication: A Systematic Review , 2006, J. Am. Medical Informatics Assoc..

[33]  C. K. Mertz,et al.  Bringing meaning to numbers: the impact of evaluative categories on decisions. , 2009, Journal of experimental psychology. Applied.

[34]  Thomas M. Hamill,et al.  Ensemble Reforecasting: Improving Medium-Range Forecast Skill Using Retrospective Forecasts , 2004 .

[35]  Clemens Simmer,et al.  HErZ: The German Hans-Ertel Centre for Weather Research , 2016 .

[36]  ZOE HILDON,et al.  Impact of format and content of visual display of data on comprehension, choice and preference: a systematic review. , 2012, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[37]  Thorsten Pachur,et al.  Interpreting Uncertainty: A Brief History of Not Knowing , 2019, Taming Uncertainty.

[38]  Antonio Maldonado,et al.  Improving risk understanding across ability levels: Encouraging active processing with dynamic icon arrays. , 2015, Journal of experimental psychology. Applied.

[39]  Ralph Hertwig,et al.  Risk preference shares the psychometric structure of major psychological traits , 2017, Science Advances.

[40]  Gerd Gigerenzer,et al.  Communicating Statistical Information , 2000, Science.

[41]  Tim N. Palmer,et al.  The economic value of ensemble forecasts as a tool for risk assessment: From days to decades , 2002 .

[42]  G. Gigerenzer,et al.  Simple tools for understanding risks: from innumeracy to insight , 2003, BMJ : British Medical Journal.

[43]  Adrian E. Raftery,et al.  Use and communication of probabilistic forecasts , 2014, Stat. Anal. Data Min..

[44]  J. Mayr,et al.  Somewhere Over the Rainbow: How to Make Effective Use of Colors in Meteorological Visualizations , 2015 .

[45]  David Borland,et al.  Rainbow Color Map (Still) Considered Harmful , 2007, IEEE Computer Graphics and Applications.

[46]  Brian J Scholl,et al.  Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias , 2012, Psychonomic bulletin & review.

[47]  Lisa M. Schwartz,et al.  PSYCHOLOGICAL SCIENCE IN THE PUBLIC INTEREST Helping Doctors and Patients Make Sense of Health Statistics , 2022 .

[48]  E. Ebert Ability of a Poor Man's Ensemble to Predict the Probability and Distribution of Precipitation , 2001 .

[49]  F. Molteni,et al.  The ECMWF Ensemble Prediction System: Methodology and validation , 1996 .

[50]  Z. B. Bouallègue,et al.  Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries , 2011 .

[51]  Ralph Hertwig,et al.  The interpretation of uncertainty in ecological rationality , 2019, Synthese.

[52]  Eugenia Kalnay,et al.  Ensemble Forecasting at NMC: The Generation of Perturbations , 1993 .

[53]  Andrea Michiorri,et al.  Forecasting for dynamic line rating , 2015 .

[54]  Jon Gill,et al.  Communicating Forecast Uncertainty for service providers , 2007 .

[55]  Geoff Cumming,et al.  Inference by Eye: Pictures of Confidence Intervals and Thinking About Levels of Confidence , 2007 .

[56]  Susan Joslyn,et al.  The Cry Wolf Effect and Weather‐Related Decision Making , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[57]  Elke U. Weber,et al.  Contextual Effects in the Interpretations of Probability Words: Perceived Base Rate and Severity of Events , 1990 .

[58]  J. Hausman Contingent Valuation: From Dubious to Hopeless , 2012 .

[59]  David S. Richardson,et al.  Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size , 2001 .

[60]  E. Kalnay,et al.  Ensemble Forecasting at NCEP and the Breeding Method , 1997 .

[61]  Bang Wong,et al.  Points of view: Mapping quantitative data to color , 2012, Nature Methods.

[62]  Dirk U. Wulff,et al.  A meta-analytic review of two modes of learning and the description-experience gap. , 2017, Psychological bulletin.

[63]  P. Pinson,et al.  Trading Wind Generation From Short-Term Probabilistic Forecasts of Wind Power , 2007, IEEE Transactions on Power Systems.

[64]  Pierre Pinson,et al.  Generation of Scenarios from Calibrated Ensemble Forecasts with a Dual-Ensemble Copula-Coupling Approach , 2015, 1511.05877.

[65]  R McGill,et al.  Graphical Perception and Graphical Methods for Analyzing Scientific Data , 1985, Science.

[66]  Antonio Maldonado,et al.  Biasing and debiasing health decisions with bar graphs: Costs and benefits of graph literacy , 2018 .

[67]  Rebecca E. Morss,et al.  Communicating Uncertainty in Weather Forecasts: A Survey of the U.S. Public , 2008 .

[68]  Susan Joslyn,et al.  Probability or frequency? Expressing forecast uncertainty in public weather forecasts , 2009 .

[69]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[70]  Steven D. Levitt,et al.  FIELD EXPERIMENTS IN ECONOMICS : THE PAST , THE PRESENT , AND THE FUTURE , 2008 .

[71]  Cynthia A. Brewer,et al.  ColorBrewer.org: An Online Tool for Selecting Colour Schemes for Maps , 2003 .

[72]  Florian Pappenberger,et al.  Challenges in communicating and using ensembles in operational flood forecasting , 2010 .

[73]  Jan K. Woike,et al.  Description and experience: How experimental investors learn about booms and busts affects their financial risk taking , 2016, Cognition.

[74]  R. Carson Contingent Valuation: A Practical Alternative When Prices Aren't Available , 2012 .

[75]  Bri-Mathias Hodge,et al.  Towards Improved Understanding of the Applicability of Uncertainty Forecasts in the Electric Power Industry , 2017 .

[76]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[77]  A. H. Murphy,et al.  A Note on the Utility of Probabilistic Predictions and the Probability Score in the Cost-Loss Ratio Decision Situation , 1966 .

[78]  O. N. Rood,et al.  ON COLOR BLINDNESS. , 1898, Science.

[79]  Mathias Zirkelbach,et al.  Critical weather situations for renewable energies – Part A: Cyclone detection for wind power , 2017 .

[80]  Michael Smithson,et al.  The interpretation of IPCC probabilistic statements around the world , 2014 .

[81]  Tim N. Palmer,et al.  Extended‐range predictions with ECMWF models: Time‐lagged ensemble forecasting , 1990 .

[82]  R. Buizza,et al.  A Comparison of the ECMWF, MSC, and NCEP Global Ensemble Prediction Systems , 2005 .

[83]  A. H. Murphy,et al.  The Value of Climatological, Categorical and Probabilistic Forecasts in the Cost-Loss Ratio Situation , 1977 .

[84]  Gerd Gigerenzer,et al.  Communicating Relative Risk Changes with Baseline Risk , 2014, Medical decision making : an international journal of the Society for Medical Decision Making.

[85]  Z. B. Bouallègue,et al.  Accounting for initial condition uncertainties in COSMO‐DE‐EPS , 2012 .

[86]  A. H. Murphy,et al.  The Early History of Probability Forecasts: Some Extensions and Clarifications , 1998 .

[87]  Paul K. J. Han,et al.  Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers , 2013, BMC Medical Informatics and Decision Making.

[88]  Vladimiro Miranda,et al.  Application of probabilistic wind power forecasting in electricity markets , 2013 .

[89]  Thomas Kox,et al.  Perception and use of uncertainty in severe weather warnings by emergency services in Germany , 2015 .

[90]  Peter Bauer,et al.  The quiet revolution of numerical weather prediction , 2015, Nature.

[91]  J. Frick,et al.  Can end-users' flood management decision making be improved by information about forecast uncertainty? , 2011 .

[92]  Tobias Pardowitz,et al.  Human estimates of warning uncertainty : Numerical and verbal descriptions , 2015 .

[93]  Yves-Marie Saint-Drenan,et al.  Critical weather situations for renewable energies – Part B: Low stratus risk for solar power , 2017 .