A bricolage-style exploratory scenario analysis to manage uncertainty in socio-environmental systems modeling: investigating integrated water management options

Exploratory analysis, while useful in assessing the implications of model assumptions under large uncertainty, is considered at best a semi-structured activity. There is no algorithmic way for performing exploratory analysis and the existing canonical techniques have their own limitations. To overcome this, we advocate a bricolage-style exploratory scenario analysis, which can be crafted by pragmatically and strategically combining different methods and practices. Our argument is illustrated using a case study in integrated water management in the Murray-Darling Basin, Australia. Scenario ensembles are generated to investigate potential policy innovations, climate and crop market conditions, as well as the effects of uncertainties in model components and parameters. Visualizations, regression trees and marginal effect analyses are exploited to make sense of the ensemble of scenarios. The analysis includes identifying patterns within a scenario ensemble, by visualizing initial hypotheses that are informed by prior knowledge, as well as by visualizing new hypotheses based on identified influential variables. Context-specific relationships are explored by analyzing which values of drivers and management options influence outcomes. Synthesis is achieved by identifying context-specific solutions to consider as part of policy design. The process of analysis is cast as a process of finding patterns and formulating questions within the ensemble of scenarios that merit further examination, allowing end-users to make the decision as to what underlying assumptions should be accepted, and whether uncertainties have been sufficiently explored. This approach is especially advantageous when the precise intentions of management are still subject to deliberations. By describing the reasoning and steps behind a bricolage-style exploratory analysis, we hope to raise awareness of the value of sharing this kind of (common but not often documented) analysis process, and motivate further work to improve sharing of know-how about bricolage in practice.

[1]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[2]  Klaus Keller,et al.  Robust Climate Policies Under Uncertainty: A Comparison of Robust Decision Making and Info‐Gap Methods , 2012, Risk analysis : an official publication of the Society for Risk Analysis.

[3]  Vijay K. Vaishnavi,et al.  Using Patterns to Illuminate Research Practice , 2007 .

[4]  James S. Hodges,et al.  Is It You or Your Model Talking?: A Framework for Model Validation , 1992 .

[5]  A. Jakeman,et al.  Computation of the instantaneous unit hydrograph and identifiable component flows with application to two small upland catchments , 1990 .

[6]  Alexis Tsoukiàs,et al.  From decision theory to decision aiding methodology , 2008, Eur. J. Oper. Res..

[7]  Myles T. Collins,et al.  Managing the Risk of Uncertain Threshold Responses: Comparison of Robust, Optimum, and Precautionary Approaches , 2007, Risk analysis : an official publication of the Society for Risk Analysis.

[8]  E. Tufte,et al.  The visual display of quantitative information , 1984, The SAGE Encyclopedia of Research Design.

[9]  A. Boulesteix,et al.  Bias in random forest variable importance measures: Illustrations, sources and a solution , 2007, BMC Bioinformatics.

[10]  Barry Croke,et al.  The development of a simple model to investigate the impact of groundwater extraction on river flows in the Namoi Catchment, NSW, Australia , 2005 .

[11]  Berry Gersonius,et al.  Accounting for uncertainty and flexibility in flood risk management: comparing Real‐In‐Options optimisation and Adaptation Tipping Points , 2015 .

[12]  M. V. Asselt,et al.  Uncertainty in perspective , 1996 .

[13]  Jan H. Kwakkel,et al.  PyNetLogo: Linking NetLogo with Python , 2018, J. Artif. Soc. Soc. Simul..

[14]  Robert J. Lempert,et al.  Comparing Algorithms for Scenario Discovery , 2008 .

[15]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[16]  Jeffrey G. Arnold,et al.  Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations , 2007 .

[17]  F. Pappenberger,et al.  Ignorance is bliss: Or seven reasons not to use uncertainty analysis , 2006 .

[18]  Joseph R. Kasprzyk,et al.  Many objective robust decision making for complex environmental systems undergoing change , 2012, Environ. Model. Softw..

[19]  Steven C Bankes,et al.  Tools and techniques for developing policies for complex and uncertain systems , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Kyle R. Douglas-Mankin,et al.  Hydrologic and Water Quality Models: Use, Calibration, and Validation , 2012 .

[21]  Anthony J. Jakeman,et al.  A catchment moisture deficit module for the IHACRES rainfall-runoff model , 2004, Environ. Model. Softw..

[22]  Moreno Marzolla,et al.  New trends in parallel and distributed simulation: From many-cores to Cloud Computing , 2014, Simul. Model. Pract. Theory.

[23]  Jennifer Badham,et al.  Effective modeling for Integrated Water Resource Management: A guide to contextual practices by phases and steps and future opportunities , 2019, Environ. Model. Softw..

[24]  Andrea Castelletti,et al.  Many‐objective reservoir policy identification and refinement to reduce policy inertia and myopia in water management , 2014 .

[25]  Benjamin P. Bryant,et al.  Thinking Inside the Box , 2010 .

[26]  Jan H. Kwakkel,et al.  How robust is a robust policy? A comparative analysis of alternative robustness metrics for supporting robust decision analysis , 2015 .

[27]  Warren E. Walker,et al.  Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty , 2013 .

[28]  Joseph H. A. Guillaume,et al.  An iterative method for discovering feasible management interventions and targets conjointly using uncertainty visualizations , 2015, Environ. Model. Softw..

[29]  A. Jakeman,et al.  How much complexity is warranted in a rainfall‐runoff model? , 1993 .

[30]  Anthony J. Jakeman,et al.  The influence of model simplicity on uncertainty in the context of surface - groundwater modelling and integrated assessment , 2011 .

[31]  J. Kincheloe On to the Next Level: Continuing the Conceptualization of the Bricolage , 2005 .

[32]  Steve Bankes,et al.  Exploratory Modeling for Policy Analysis , 1993, Oper. Res..

[33]  Byron K Williams,et al.  Passive and active adaptive management: approaches and an example. , 2011, Journal of environmental management.

[34]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[35]  Jonathan D. Herman,et al.  How should robustness be defined for water systems planning under change , 2015 .

[36]  J. Hernandez,et al.  Uncertainty Considerations in Calibration and Validation of Hydrologic and Water Quality Models , 2015 .

[37]  Jan H. Kwakkel,et al.  The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making , 2017, Environ. Model. Softw..

[38]  Tuomas J. Lahtinen,et al.  Why pay attention to paths in the practice of environmental modelling? , 2017, Environ. Model. Softw..

[39]  Joseph H. A. Guillaume,et al.  Robust discrimination between uncertain management alternatives by iterative reflection on crossover point scenarios: Principles, design and implementations , 2016, Environ. Model. Softw..

[40]  Hadley Wickham,et al.  ggplot2 - Elegant Graphics for Data Analysis (2nd Edition) , 2017 .

[41]  Kerrylee Rogers,et al.  Floodplain Wetland Biota in the Murray-Darling Basin , 2010 .

[42]  Wolfgang Weimer-Jehle,et al.  Cross-impact balances: A system-theoretical approach to cross-impact analysis , 2006 .

[43]  Casey Brown,et al.  Decision scaling: Linking bottom‐up vulnerability analysis with climate projections in the water sector , 2012 .

[44]  Achim Zeileis,et al.  Bias in random forest variable importance measures: Illustrations, sources and a solution , 2007, BMC Bioinformatics.

[45]  Muhittin Oral,et al.  Model legitimisation in operational research , 1996 .

[46]  Francis H. S. Chiew,et al.  Use of seasonal streamflow forecasts in water resources management , 2003 .

[47]  J. Kincheloe Describing the Bricolage: Conceptualizing a New Rigor in Qualitative Research , 2001 .

[48]  Andrea Castelletti,et al.  Is robustness really robust? How different definitions of robustness impact decision-making under climate change , 2016, Climatic Change.

[49]  Jenifer Lyn Ticehurst,et al.  Can existing practices expected to lead to improved on-farm water use efficiency enable irrigators to effectively respond to reduced water entitlements in the Murray-Darling Basin? , 2015 .

[50]  Gail Clement,et al.  Toward the Geoscience Paper of the Future: Best practices for documenting and sharing research from data to software to provenance , 2017 .

[51]  Zoran Kapelan,et al.  Comparison of Robust Optimization and Info-Gap Methods for Water Resource Management under Deep Uncertainty , 2016 .

[52]  Warren E. Walker,et al.  Developing dynamic adaptive policy pathways: a computer-assisted approach for developing adaptive strategies for a deeply uncertain world , 2015, Climatic Change.

[53]  Robert J Lempert,et al.  A new decision sciences for complex systems , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[54]  Patrick M. Reed,et al.  An open source framework for many-objective robust decision making , 2015, Environ. Model. Softw..

[55]  Joseph H. A. Guillaume,et al.  Hypothesis Testing for Management: Evolving and Answering Closed Questions Using Multiobjective Visualization , 2014 .

[56]  Jan H. Kwakkel,et al.  Supporting DMDU: A Taxonomy of Approaches and Tools , 2019, Decision Making under Deep Uncertainty.

[57]  Geoffrey Paterson,et al.  Guide to the Proposed Basin Plan: Overview ; Guide to the Proposed Basin Plan: Technical Background [Book Review] , 2011 .

[58]  Download Book,et al.  Information Visualization in Data Mining and Knowledge Discovery , 2001 .

[59]  E. A. Moallemi,et al.  Model-based multi-objective decision making under deep uncertainty from a multi-method design lens , 2018, Simul. Model. Pract. Theory.

[60]  Peng Yue,et al.  Model provenance tracking and inference for integrated environmental modelling , 2017, Environ. Model. Softw..

[61]  Naresh Pai,et al.  A Recommended Calibration and Validation Strategy for Hydrologic and Water Quality Models , 2015 .

[62]  Yao Hu,et al.  Global sensitivity analysis for large-scale socio-hydrological models using Hadoop , 2015, Environ. Model. Softw..

[63]  Helwig Hauser,et al.  Visualization and Visual Analysis of Multifaceted Scientific Data: A Survey , 2013, IEEE Transactions on Visualization and Computer Graphics.

[64]  K. Reinhardt Shaping The Next One Hundred Years New Methods For Quantitative Long Term Policy Analysis , 2016 .

[65]  J. Harou,et al.  Selecting Portfolios of Water Supply and Demand Management Strategies Under Uncertainty—Contrasting Economic Optimisation and ‘Robust Decision Making’ Approaches , 2013, Water Resources Management.

[66]  Warren E. Walker,et al.  Comparing Robust Decision-Making and Dynamic Adaptive Policy Pathways for model-based decision support under deep uncertainty , 2016, Environ. Model. Softw..

[67]  Holger R. Maier,et al.  Adaptive, multiobjective optimal sequencing approach for urban water supply augmentation under deep uncertainty , 2015 .

[68]  Joseph H. A. Guillaume,et al.  Assessing certainty and uncertainty in riparian habitat suitability models by identifying parameters with extreme outputs , 2014, Environ. Model. Softw..

[69]  Jan H. Kwakkel,et al.  Exploratory Modeling and Analysis, an approach for model-based foresight under deep uncertainty , 2013 .

[70]  Andrea Castelletti,et al.  Robustness Metrics: How Are They Calculated, When Should They Be Used and Why Do They Give Different Results? , 2018 .

[71]  Joseph H. A. Guillaume,et al.  Prediction under uncertainty as a boundary problem: A general formulation using Iterative Closed Question Modelling , 2015, Environ. Model. Softw..

[72]  Vladan Babovic,et al.  Adaptation Pathways and Real Options Analysis: An approach to deep uncertainty in climate change adaptation policies , 2016 .

[73]  Zoran Kapelan,et al.  Comparison of Info-gap and Robust Optimisation Methods for Integrated Water Resource Management under Severe Uncertainty☆ , 2015 .

[74]  J. Kwadijk,et al.  Using adaptation tipping points to prepare for climate change and sea level rise: a case study in the Netherlands , 2010 .

[75]  P. Schoemaker Scenario Planning: A Tool for Strategic Thinking , 1995 .

[76]  John Norton,et al.  An introduction to sensitivity assessment of simulation models , 2015, Environ. Model. Softw..

[77]  Dharmendra Saraswat,et al.  Hydrologic and Water Quality Models: Key Calibration and Validation Topics , 2015 .

[78]  J. Refsgaard,et al.  Operational Validation and Intercomparison of Different Types of Hydrological Models , 1996 .

[79]  Joseph H. A. Guillaume,et al.  An uncertain future, deep uncertainty, scenarios, robustness and adaptation: How do they fit together? , 2016, Environ. Model. Softw..

[80]  Jan H. Kwakkel,et al.  Adaptive Robust Design under deep uncertainty , 2013 .

[81]  Anthony J. Jakeman,et al.  Model development for integrated assessment of water allocation options , 2004 .

[82]  Baihua Fu,et al.  A note on communicating environmental change for non-market valuation , 2017 .

[83]  Jan H. Kwakkel,et al.  How Robust is a Robust Policy? Comparing Alternative Robustness Metrics for Robust Decision-Making , 2016 .

[84]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[85]  David Hadka High Performance Computing with the MOEA Framework and Ignite , 2016 .

[86]  K. Hornik,et al.  Unbiased Recursive Partitioning: A Conditional Inference Framework , 2006 .

[87]  Joseph H. A. Guillaume,et al.  Fit for purpose? Building and evaluating a fast, integrated model for exploring water policy pathways , 2014, Environ. Model. Softw..

[88]  Peter A. Vanrolleghem,et al.  Uncertainty in the environmental modelling process - A framework and guidance , 2007, Environ. Model. Softw..

[89]  Joyce S. R. Yee Methodological Innovation in Practice-Based Design Doctorates , 2010 .

[90]  G. De’ath,et al.  CLASSIFICATION AND REGRESSION TREES: A POWERFUL YET SIMPLE TECHNIQUE FOR ECOLOGICAL DATA ANALYSIS , 2000 .

[91]  Anthony J. Jakeman,et al.  Integrated assessment of water resources: Australian experiences , 2006 .

[92]  Mark E. Borsuk,et al.  Does high forecast uncertainty preclude effective decision support? , 2005, Environ. Model. Softw..

[93]  James S. Hodges,et al.  Six (Or So) Things You Can Do with a Bad Model , 1991, Oper. Res..