Communicating model uncertainty for natural hazards: A qualitative systematic thematic review

Abstract Natural hazard models are vital for all phases of risk assessment and disaster management. However, the high number of uncertainties inherent to these models is highly challenging for crisis communication. The non-communication of these is problematic as interdependencies between them, especially for multi-model approaches and cascading hazards, can result in much larger deep uncertainties. The recent upsurge in research into uncertainty communication makes it important to identify key lessons, areas for future development, and areas for future research. We present a systematic thematic literature review to identify methods for effective communication of model uncertainty. Themes identified include a) the need for clear uncertainty typologies, b) the need for effective engagement with users to identify which uncertainties to focus on, c) managing ensembles, confidence, bias, consensus and dissensus, d) methods for communicating specific uncertainties (e.g., maps, graphs, and time), and e) the lack of evaluation of many approaches currently in use. Finally, we identify lessons and areas for future investigation, and propose a framework to manage the communication of model related uncertainty with decision-makers, by integrating typology components that help identify and prioritise uncertainties. We conclude that scientists must first understand decision-maker needs, and then concentrate efforts on evaluating and communicating the decision-relevant uncertainties. Developing a shared uncertainty management scheme with users facilitates the management of different epistemological perspectives, accommodates the different values that underpin model assumptions and the judgements they prompt, and increases uncertainty tolerance. This is vital, as uncertainties will only increase as our model (and event) complexities increase.

[1]  Katharina Helming,et al.  Socially responsible research processes for sustainability transformation: an integrated assessment framework , 2016 .

[2]  P. Dickmann,et al.  Making sense of communication interventions in public health emergencies – an evaluation framework for risk communication , 2015 .

[3]  S. Shackley,et al.  Representing Uncertainty in Global Climate Change Science and Policy: Boundary-Ordering Devices and Authority , 1996 .

[4]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[5]  Matthew Thompson,et al.  Natural Hazard Modeling and Uncertainty Analysis , 2016 .

[6]  Mark S Jackson Risk and Uncertainty Assessment for Natural Hazards , 2013 .

[7]  Kevin D. Ash,et al.  Tornado warning trade-offs: evaluating choices for visually communicating risk , 2014 .

[8]  Paul K J Han,et al.  Conceptual, Methodological, and Ethical Problems in Communicating Uncertainty in Clinical Evidence , 2013, Medical care research and review : MCRR.

[9]  Douglas Paton,et al.  Decision-Making: Preventing Miscommunication and Creating Shared Meaning Between Stakeholders , 2017 .

[10]  E. Markowitz,et al.  The moral complexity of climate change and the need for a multidisciplinary perspective on climate ethics , 2015, Climatic Change.

[11]  Mark Petticrew,et al.  Systematic reviews from astronomy to zoology: myths and misconceptions , 2001, BMJ : British Medical Journal.

[12]  Gabriele Bammer,et al.  Uncertainty and Risk: Multidisciplinary Perspectives , 2008 .

[13]  R. Lipshitz,et al.  Coping with Uncertainty: A Naturalistic Decision-Making Analysis , 1997 .

[14]  Joseph Mendola,et al.  From the Good , 2014 .

[15]  Helen M. Regan,et al.  Mapping epistemic uncertainties and vague concepts in predictions of species distribution , 2002 .

[16]  S. Broomell,et al.  Effective communication of uncertainty in the IPCC reports , 2012, Climatic Change.

[17]  Jerome R. Ravetz,et al.  The good, the true and the post-modern , 1992 .

[18]  Jerome R. Ravetz,et al.  Uncertainty and Quality in Science for Policy , 1990 .

[19]  Robert L Winkler,et al.  The Importance of Communicating Uncertainties in Forecasts: Overestimating the Risks from Winter Storm Juno , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[20]  Bruna De Marchi,et al.  When the earth shakes … and science with it. The management and communication of uncertainty in the L’Aquila earthquake , 2017 .

[21]  Chris R. Johnson,et al.  A Next Step: Visualizing Errors and Uncertainty , 2003, IEEE Computer Graphics and Applications.

[22]  Alan M. MacEachren,et al.  Evaluating the effect of visually represented geodata uncertainty on decision-making: systematic review, lessons learned, and recommendations , 2017 .

[23]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[24]  Shardul Agrawala,et al.  Climate change. The limits of consensus. , 2007, Science.

[25]  L. Lemyre,et al.  Public Reactions to Risk Messages Communicating Different Sources of Uncertainty: An Experimental Test , 2013 .

[26]  Mark Gahegan,et al.  A typology for visualizing uncertainty , 2005, IS&T/SPIE Electronic Imaging.

[27]  Hayley J. Fowler,et al.  Examination of climate risk using a modified uncertainty matrix framework—Applications in the water sector , 2013 .

[28]  Erwin Fielt,et al.  European Conference on Information Systems ( ECIS ) Summer 10-6-2011 A SYSTEMATIC , TOOL-SUPPORTED METHOD FOR CONDUCTING LITERATURE REVIEWS IN INFORMATION SYSTEMS , 2017 .

[29]  Steven P. French,et al.  Risk Assessment, Modeling and Decision Support , 2008 .

[30]  Arthur C. Petersen,et al.  A method for the analysis of assumptions in model-based environmental assessments , 2011, Environ. Model. Softw..

[31]  John Handmer,et al.  Communicating uncertainty via probabilities: The case of weather forecasts , 2007 .

[32]  Andrew N Freedman,et al.  Laypersons' Responses to the Communication of Uncertainty Regarding Cancer Risk Estimates , 2009, Medical decision making : an international journal of the Society for Medical Decision Making.

[33]  Kirsten J McCaffery,et al.  The effects of communicating uncertainty in quantitative health risk estimates. , 2012, Patient education and counseling.

[34]  Douglas Paton,et al.  Organisational response to a volcanic eruption , 1998 .

[35]  Elias Ioup,et al.  Communicating uncertainty information across conceptual boundaries , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[36]  Paul Slovic,et al.  Lay views on uncertainty in environmental health risk assessment , 1998 .

[37]  Pricilla Marimo,et al.  Communication of Uncertainty in Temperature Forecasts , 2015 .

[38]  Silvio Funtowicz,et al.  Post-Normal Science — an insight now maturing , 1999 .

[39]  M Sandelowski,et al.  Qualitative Metasynthesis : Issues and Techniques , 1997 .

[40]  Susan Miles,et al.  Public perception of scientific uncertainty in relation to food hazards , 2003 .

[41]  Thomas J. Chermack,et al.  Using Scenarios to Develop Crisis Managers: Applications of Scenario Planning and Scenario-Based Training , 2008 .

[42]  Alberto Cairo,et al.  Graphics Lies, Misleading Visuals , 2015 .

[43]  Alex T. Pang,et al.  Approaches to uncertainty visualization , 1996, The Visual Computer.

[44]  Milind Kandlikar,et al.  Expressions of likelihood and confidence in the IPCC uncertainty assessment process , 2007 .

[45]  Jim W. Hall,et al.  Epistemic uncertainties and natural hazard risk assessment ­ Part 1: A review of the issues , 2015 .

[46]  Andrew Stirling,et al.  Keep it complex , 2010, Nature.

[47]  Stefan Leyk,et al.  A Conceptual Framework for Uncertainty Investigation in Map‐based Land Cover Change Modelling , 2005, Trans. GIS.

[48]  Mike Pearson,et al.  Visualizing Uncertainty About the Future , 2022 .

[49]  Lucy Bastin,et al.  Visualizing uncertainty in multi-spectral remotely sensed imagery , 2002 .

[50]  Paul Slovic,et al.  The Effects of Presenting Imprecise Probabilities in Intelligence Forecasts , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[51]  J. Byrne,et al.  The benefits of publishing systematic quantitative literature reviews for PhD candidates and other early-career researchers , 2014 .

[52]  Johannes Cullmann,et al.  Communicating flood forecast uncertainty under operational circumstances , 2009 .

[53]  T. Wilson,et al.  Integrating multidisciplinary science, modelling and impact data into evolving, syn-event volcanic hazard mapping and communication: A case study from the 2012 Tongariro eruption crisis, New Zealand , 2014 .

[54]  Wolfgang Raskob,et al.  Approaches to visualisation of uncertainties to decision makers in an operational Decision Support System , 2009 .

[55]  David Johnston,et al.  Knowledge transfer between communities, practitioners, and researchers: A case study for community resilience in Wellington, New Zealand , 2015 .

[56]  Jonathan Karnon,et al.  Model Parameter Estimation and Uncertainty: a Report of the Ispor-smdm Modeling Good Research Practices Task Force-6 Background to the Task Forcemodel-parameter-estimation-and- Uncertainty-analysis.asp). a Summary of These Articles Was Pre- Sented at a Plenary Session at the Ispor 16th Annual Intern , 2022 .

[57]  Guy H. Walker,et al.  Measuring Situation Awareness in complex systems: Comparison of measures study , 2009 .

[58]  Irem Y. Tumer,et al.  The Capture, Assessment and Communication Tool for Uncertainty Simulation (CACTUS) in Complex Systems , 2008 .

[59]  C. Pickering,et al.  Are Weeds Hitchhiking a Ride on Your Car? A Systematic Review of Seed Dispersal on Cars , 2013, PloS one.

[60]  H. Griethe Visualizing Uncertainty for Improved Decision Making , 2005 .

[61]  M. Gibbons,et al.  Re-Thinking Science: Knowledge and the Public in an Age of Uncertainty , 2003 .

[62]  W. Walker,et al.  Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support , 2003 .

[63]  J. Rougier,et al.  Risk and Uncertainty Assessment for Natural Hazards: Quantifying hazard losses , 2013 .

[64]  M W Blind,et al.  Operationalising uncertainty in data and models for integrated water resources management. , 2007, Water science and technology : a journal of the International Association on Water Pollution Research.

[65]  O. O’neill,et al.  Autonomy and Trust in Bioethics: Autonomy and Trust in Bioethics , 2002 .

[66]  J. Byrne,et al.  Past results and future directions in urban community gardens research , 2012 .

[67]  J. A. Wardekker,et al.  Uncertainty Communication : Issues and good practice , 2007 .

[68]  Wendy S. Parker,et al.  Predicting weather and climate: Uncertainty, ensembles and probability , 2010 .

[69]  S. Funtowicz,et al.  Science for the Post-Normal Age , 1993, Commonplace.

[70]  Joe Michael Kniss,et al.  Visualizing Summary Statistics and Uncertainty , 2010, Comput. Graph. Forum.

[71]  David Retchless,et al.  Sea Level Rise Maps: How Individual Differences Complicate the Cartographic Communication of an Uncertain Climate Change Hazard , 2014 .

[72]  Keith Beven,et al.  Equipped to deal with uncertainty in climate and impacts predictions: lessons from internal peer review , 2015, Climatic Change.

[73]  Anthony Patt,et al.  Assessing model-based and conflict-based uncertainty , 2007 .

[74]  Wendy S. Parker,et al.  Ensemble modeling, uncertainty and robust predictions , 2013 .

[75]  Rebecca M. B. Harris,et al.  A tool for simulating and communicating uncertainty when modelling species distributions under future climates , 2014, Ecology and evolution.

[76]  David Johnston,et al.  Communicating likelihoods and probabilities in forecasts of volcanic eruptions , 2014 .

[77]  Roger M. Cooke,et al.  Special issue on expert judgment , 2008, Reliab. Eng. Syst. Saf..

[78]  S. Ferson,et al.  A Frequency/Consequence‐based Technique for Visualizing and Communicating Uncertainty and Perception of Risk , 2008, Annals of the New York Academy of Sciences.

[79]  Susan Joslyn,et al.  Methods for Communicating the Complexity and Uncertainty of Oil Spill Response Actions and Tradeoffs , 2015 .

[80]  Robert J. Moorhead,et al.  Visualizing uncertainty of river model ensembles , 2015, Electronic Imaging.

[81]  Leigh M. Smith,et al.  Volcanic Hazards: Risk Perception and Preparedness , 2000 .

[82]  Jared E. LeClerc,et al.  Climate Projections and Uncertainty Communication , 2016, Top. Cogn. Sci..

[83]  Clare E. Bond,et al.  Knowledge transfer in a digital world: field data acquisition, uncertainty, visualization, and data management , 2007 .

[84]  Herbert H. Bell,et al.  Training Decision Makers for Complex Environments: Implications of the Naturalistic Decision Making Perspective , 1997 .

[85]  Willy P Aspinall,et al.  Structured elicitation of expert judgment for probabilistic hazard and risk assessment in volcanic eruptions , 2006 .

[86]  Gary E. Bolton,et al.  A Laboratory Study of the Benefits of Including Uncertainty Information in Weather Forecasts , 2006 .

[87]  Philippe C. Baveye,et al.  Uncertainties in environmental modelling and consequences for policy making , 2009 .

[88]  Anthony Leiserowitz,et al.  Misinterpretations of the “Cone of Uncertainty” in Florida during the 2004 Hurricane Season , 2007 .

[89]  B. Fischhoff,et al.  COMMUNICATING RISK TO THE PUBLIC , 1992 .

[90]  Nick Bearman,et al.  Using sound to represent uncertainty in UKCP09 data with Google Maps API , 2013 .

[91]  Richard L Street,et al.  The importance of communication in collaborative decision making: facilitating shared mind and the management of uncertainty. , 2011, Journal of evaluation in clinical practice.

[92]  V. Johnson,et al.  Evaluations of disaster education programs for children: A methodological review , 2014 .

[93]  Marjolijn Haasnoot,et al.  Communicating climate (change) uncertainties: simulation games as boundary objects , 2015 .

[94]  D. Paton,et al.  Preparing for Disaster: Building Household and Community Capacity , 2013 .

[95]  Michael J. Brennan,et al.  Operational Forecaster Uncertainty Needs and Future Roles , 2008 .

[96]  Kevin J. Shanahan,et al.  Scenario Planning: Toward a More Complete Model for Practice , 2008 .

[97]  Paolo Gardoni,et al.  Classification and Moral Evaluation of Uncertainties in Engineering Modeling , 2011, Sci. Eng. Ethics.

[98]  Limor Nadav-Greenberg,et al.  The effects of wording on the understanding and use of uncertainty information in a threshold forecasting decision , 2009 .

[99]  M. Oppenheimer,et al.  The ethics of scientific communication under uncertainty , 2014 .

[100]  P. Schmidt-Thomé,et al.  Communicating uncertainty in climate-change adaptation and decision support; further development of the Gdańsk case study , 2008 .

[101]  David B. Portnoy,et al.  Physicians’ attitudes about communicating and managing scientific uncertainty differ by perceived ambiguity aversion of their patients , 2013, Health expectations : an international journal of public participation in health care and health policy.

[102]  Jean Ann Summers,et al.  Understanding Qualitative Metasynthesis , 2011 .

[103]  Keith Beven,et al.  Communicating uncertainty in flood inundation mapping: a case study , 2015 .

[104]  Terje Aven,et al.  An Evaluation of the Treatment of Risk and Uncertainties in the IPCC Reports on Climate Change , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[105]  Luca Delle Monache,et al.  A Weather and Climate Enterprise Strategic Implementation Plan for Generating and Communicating Forecast Uncertainty Information , 2011 .

[106]  David J. Duke,et al.  Uncertainty visualization: why might it fail? , 2009, CHI Extended Abstracts.

[107]  Patrick Willems,et al.  A non-parametric data-based approach for probabilistic flood forecasting in support of uncertainty communication , 2012, Environ. Model. Softw..

[108]  Willy P Aspinall,et al.  Comment on “Social studies of volcanology: knowledge generation and expert advice on active volcanoes” by Amy Donovan, Clive Oppenheimer and Michael Bravo [Bull Volcanol (2012) 74:677-689] , 2012, Bulletin of Volcanology.

[109]  Anthony Patt,et al.  Using Specific Language to Describe Risk and Probability , 2003 .

[111]  Michael Gibbons,et al.  Introduction: `Mode 2' Revisited: The New Production of Knowledge , 2003 .

[112]  Keith Beven,et al.  Model and data limitations:The sources and implications of epistemic uncertainty , 2013 .

[113]  David Johnston,et al.  Enhancing scientific response in a crisis: evidence-based approaches from emergency management in New Zealand , 2015, Journal of Applied Volcanology.

[114]  Bernd Rohrmann,et al.  The evaluation of risk communication effectiveness , 1992 .

[115]  Pamela J. Bretschneider,et al.  The future that may (or may not) come: How framing changes responses to uncertainty in climate change communications , 2011 .

[116]  B. Fauser,et al.  Balancing the strengths of systematic and narrative reviews. , 2005, Human reproduction update.

[117]  M. Jackson Social science perspectives on natural hazards risk and uncertainty , 2013 .

[118]  Sarb Johal,et al.  Communicating with the Public during an Earthquake Sequence: Improving Communication of Geoscience by Coordinating Roles , 2016 .

[119]  Bongshin Lee,et al.  Revealing Uncertainty for Information Visualization , 2010, Inf. Vis..

[120]  Martin J. Eppler,et al.  The Risks of Visualization: A Classification of Disadvantages Associated with Graphic Representations of Information , 2009 .

[121]  D A Stainforth,et al.  Confidence, uncertainty and decision-support relevance in climate predictions , 2007, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[122]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[123]  Brian C. O'Neill,et al.  The Limits of Consensus , 2007, Science.

[124]  David Johnston,et al.  Uncertainty and decision making: Volcanic crisis scenarios , 2014 .

[125]  D. Baker,et al.  Magnetosphere-Ionosphere Coupling in the Solar System , 2016 .

[126]  K. Mach,et al.  Understanding, characterizing, and communicating responses to ocean acidification : challenges and uncertainties , 2015 .

[127]  Diane Keeble-Ramsay,et al.  The rapid structured literature review as a research strategy , 2009 .

[128]  K. K. Cetina Culture in global knowledge societies: knowledge cultures and epistemic cultures , 2007 .

[129]  A. Edwards,et al.  Communicating risk , 2012, BMJ : British Medical Journal.

[130]  Tonya L Smith-Jackson,et al.  Research-based guidelines for warning design and evaluation. , 2002, Applied ergonomics.

[131]  Roger M. Cooke,et al.  Quantifying scientific uncertainty from expert judgement elicitation , 2013 .

[132]  Alexander Toet,et al.  Public Understanding of Visual Representations of Uncertainty in Temperature Forecasts , 2015 .

[133]  Rebecca E. Morss,et al.  Communicating Uncertainty in Weather Forecasts: A Survey of the U.S. Public , 2008 .

[134]  C. Newhall,et al.  The Narrow Margin of Successful Volcanic-Risk Mitigation , 1996 .

[135]  M. Smithson,et al.  Conflict Aversion: Preference for Ambiguity vs Conflict in Sources and Evidence. , 1999, Organizational behavior and human decision processes.

[136]  O. O’neill Autonomy and Trust in Bioethics , 2002 .

[137]  Luc Anselin,et al.  Visualizing Seismic Risk and Uncertainty , 2008, Annals of the New York Academy of Sciences.

[138]  Jutta Thielen,et al.  Ensemble predictions and perceptions of risk, uncertainty, and error in flood forecasting , 2007 .

[139]  P. Cronin,et al.  Undertaking a literature review: a step-by-step approach. , 2008, British journal of nursing.

[140]  Arnulf Grubler,et al.  Coping with uncertainties-examples of modeling approaches at IIASA☆ , 2015 .

[141]  Francis W. Zwiers,et al.  Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties , 2010 .

[142]  C. Peter Keller,et al.  Modelling and visualizing multiple spatial uncertainties , 1997 .

[143]  B. Johnson Further Notes on Public Response to Uncertainty in Risks and Science , 2003, Risk analysis : an official publication of the Society for Risk Analysis.

[144]  David V. Budescu,et al.  Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change , 2009, Psychological science.

[145]  Jill A.E. Gunn,et al.  A review of uncertainty research in impact assessment , 2015 .

[146]  W. Aspinall A route to more tractable expert advice , 2010, Nature.

[147]  A. Donovan,et al.  Reply to comment from W.P. Aspinall on “Social studies of volcanology: knowledge generation and expert advice on active volcanoes” by Amy Donovan, Clive Oppenheimer and Michael Bravo [Bull Volcanol (2012) 74:677-689] , 2012, Bulletin of Volcanology.

[148]  Warner Marzocchi,et al.  Natural Hazard Uncertainty Assessment: Modeling and Decision Support , 2016 .

[149]  James M. Boyle,et al.  A systematic literature review of empirical evidence on computer games and serious games , 2012, Comput. Educ..

[150]  Manuel Menezes de Oliveira Neto,et al.  Overview and State-of-the-Art of Uncertainty Visualization , 2014, Scientific Visualization.

[151]  M.B.A. van Asselt,et al.  Perspectives on Uncertainty and Risk: The PRIMA Approach to Decision Support , 2000 .

[152]  Maria J Grant,et al.  A typology of reviews: an analysis of 14 review types and associated methodologies. , 2009, Health information and libraries journal.

[153]  Anna Rabinovich,et al.  Unquestioned Answers or Unanswered Questions: Beliefs About Science Guide Responses to Uncertainty in Climate Change Risk Communication , 2012, Risk analysis : an official publication of the Society for Risk Analysis.

[154]  Jim W. Hall,et al.  Epistemic uncertainties and natural hazard risk assessment – Part 2: What should constitute good practice? , 2017, Natural Hazards and Earth System Sciences.

[155]  J R Ravetz,et al.  A guidance for assessing and communicating uncertainties. , 2005, Water science and technology : a journal of the International Association on Water Pollution Research.

[156]  David Bihanic New Challenges for Data Design , 2015 .

[157]  Julia Åhlén,et al.  Geovisualisation of uncertainty in simulated flood maps , 2014 .

[158]  Kenneth I. Joy,et al.  Characterizing and Visualizing Predictive Uncertainty in Numerical Ensembles Through Bayesian Model Averaging , 2013, IEEE Transactions on Visualization and Computer Graphics.

[159]  Gary J. Hunter,et al.  Testing the Effects of Thematic Uncertainty on Spatial Decision-making , 2007 .

[160]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[161]  Michael J. McCloskey,et al.  Decision skills training: Facilitating learning from experience. , 2001 .

[162]  L. Maxim,et al.  How is Scientific Credibility Affected by Communicating Uncertainty? The Case of Endocrine Disrupter Effects on Male Fertility , 2014 .

[163]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[164]  Anthony Patt,et al.  Communicating uncertainty to policy makers , 2009 .

[165]  A. Fisher,et al.  Risk Communication Challenges1 , 1991 .

[166]  Jeffrey D Myers,et al.  The Influence of Uncertain Map Features on Risk Beliefs and Perceived Ambiguity for Maps of Modeled Cancer Risk from Air Pollution , 2013, Risk analysis : an official publication of the Society for Risk Analysis.

[167]  Douglas Paton,et al.  Risk perception and volcanic hazard mitigation: Individual and social perspectives , 2008 .

[168]  Vincent Marchau,et al.  Classifying and communicating uncertainties in model-based policy analysis , 2010 .

[169]  Michael J. Bloom,et al.  Scenario Planning and Contingency Planning , 1994 .

[170]  D. Paton,et al.  Developing disaster management capability: an assessment centre approach , 2002 .

[171]  B. Johnson Risk Communication: A Mental Models Approach , 2002 .

[172]  Amy Donovan,et al.  Social studies of volcanology: knowledge generation and expert advice on active volcanoes , 2012, Bulletin of Volcanology.

[173]  Limor Nadav-Greenberg,et al.  The Effect of Uncertainty Visualizations on Decision Making in Weather Forecasting , 2008 .

[174]  P. Slovic,et al.  Presenting uncertainty in health risk assessment: initial studies of its effects on risk perception and trust. , 1995, Risk analysis : an official publication of the Society for Risk Analysis.

[175]  R. Rosenfeld Confidence , 2007, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[176]  Adrienne Grêt-Regamey,et al.  Visualization of uncertainty in natural hazards assessments using an interactive cartographic information system , 2011 .

[177]  Natalia I. Deligne,et al.  Overview of the co-ordinated risk-based approach to science and management response and recovery for the 2012 eruptions of Tongariro volcano, New Zealand , 2014 .

[178]  Mathew P. White,et al.  Risk interpretation and action: A conceptual framework for responses to natural hazards , 2012 .

[179]  Keith Beven,et al.  Developing a Translational Discourse to Communicate Uncertainty in Flood Risk between Science and the Practitioner , 2007, Ambio.

[180]  A. Patt,et al.  Communicating uncertainty: lessons learned and suggestions for climate change assessment , 2005 .

[181]  P. Wiedemann,et al.  Perception of Uncertainty and Communication about Unclear Risks , 2008 .

[182]  Christopher J. Roy,et al.  A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing , 2011 .

[183]  Richard H. Moss,et al.  Reducing doubt about uncertainty: Guidance for IPCC’s third assessment , 2011 .

[184]  V. Murray,et al.  Ensuring science is useful, usable and used in global disaster risk reduction and sustainable development: a view through the Sendai framework lens , 2016, Palgrave Communications.

[185]  Eric Winsberg Values and Uncertainties in the Predictions of Global Climate Models , 2012, Kennedy Institute of Ethics journal.

[186]  Todd Schenk,et al.  Embodied uncertainty: living with complexity and natural hazards , 2018 .

[187]  K. Lowell,et al.  Visualisation of spatial uncertainty in hydrological modelling , 2011 .

[188]  M. Maslin Commentary Cascading uncertainty in climate change models and its implications for policy , 2013 .

[189]  P. Wiedemann,et al.  Lessons Learned: Recommendations for Communicating Conflicting Evidence for Risk Characterization , 2008 .

[190]  John Paul Gosling,et al.  Building narratives to characterise uncertainty in regional climate change through expert elicitation , 2018, Environmental Research Letters.

[191]  Britta Höllermann,et al.  Integration of uncertainties in water and flood risk management , 2015 .

[192]  R. Cooke Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[193]  Gerassimos A. Papadopoulos,et al.  OPERATIONAL EARTHQUAKE FORECASTING. State of Knowledge and Guidelines for Utilization , 2011 .

[194]  Hans von Storch,et al.  Introduction: Post‐Normal Climate Science , 2012 .

[195]  Salvatore Lupo,et al.  The Capture , 2014, At the Point of a Cutlass.

[196]  Massimiliano Zappa,et al.  Visualizing flood forecasting uncertainty: some current European EPS platforms—COST731 working group 3 , 2010 .

[197]  Karl Claxton,et al.  Characterizing structural uncertainty in decision analytic models: a review and application of methods. , 2009, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[198]  Michael G. Moore,et al.  Assessment of Oral Communication: A Major Review of the Historical Development and Trends in the Movement from 1975 to 2009 , 2011 .

[199]  Alex Pang,et al.  Visualizing Uncertainty in Natural Hazards , 2008 .

[200]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[201]  Douglas Paton,et al.  Developing a comprehensive model of hazard preparedness: Lessons from the Christchurch earthquake , 2015 .

[202]  Gertrude Hirsch Hadorn,et al.  Article type: Advanced Review Article title: The IPCC and treatment of uncertainties: topics and sources of dissensus , 2014 .

[203]  E. Mohammadi,et al.  Barriers and facilitators related to the implementation of a physiological track and trigger system: A systematic review of the qualitative evidence , 2017, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[204]  Florian Pappenberger,et al.  Visualizing probabilistic flood forecast information: expert preferences and perceptions of best practice in uncertainty communication , 2011 .

[205]  Mary Anne Thompson,et al.  The influence of probabilistic volcanic hazard map properties on hazard communication , 2015, Journal of Applied Volcanology.

[206]  Daniel P. Loucks,et al.  Quantifying and Communicating Model Uncertainty for Decisionmaking in the Everglades , 2003 .

[207]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[208]  Petr Kubícek,et al.  Thematic uncertainty visualization usability – comparison of basic methods , 2011, Ann. GIS.

[209]  Jutta Thielen,et al.  Communicating uncertainty in hydrological forecasts: mission impossible? , 2010 .

[210]  Stephanie Deitrick,et al.  Developing Implicit Uncertainty Visualization Methods Motivated by Theories in Decision Science , 2015 .

[211]  Phaedra Daipha,et al.  Weathering Risk: Uncertainty, Weather Forecasting, and Expertise , 2012 .

[212]  Alexander L. Davis,et al.  Communicating scientific uncertainty , 2014, Proceedings of the National Academy of Sciences.

[213]  Alan M. MacEachren,et al.  How to Assess Visual Communication of Uncertainty? A Systematic Review of Geospatial Uncertainty Visualisation User Studies , 2014 .

[214]  Arthur C. Petersen,et al.  Reflective approaches to uncertainty assessment and communication , 2011 .

[215]  E. Borodzicz,et al.  Individual and Group Learning in Crisis Simulations , 2002 .