An agent-monitored framework for the output-oriented design of experiments in exploratory modelling

Abstract Exploratory modelling is an approach for modelling under uncertainty based on the generation and analysis of computational experiments. The results of exploratory modelling are sensitive to the way that experiments are designed, such as the way that the uncertainty space is delineated. This article introduces an agent-monitored framework—i.e. a design metaphor of the interactions among modellers and stakeholders and the simulation process—for controlling the design of experiments based on monitoring model behaviour in the output space. To demonstrate the benefits of the suggested framework in the exploratory modelling process, the article shows how the use of the framework with an output-oriented approach informs the delineation of an appropriate uncertainty space with an illustrative example in the decision-making context. The article concludes that the design of experiments based on feedback from the output space can be a useful approach: to control simulations in exploratory modelling; to build more confidence in final results; and to inform the design of other aspects of experiments, such as selecting policy levers and sampling method.

[1]  Jan H. Kwakkel,et al.  The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making , 2017, Environ. Model. Softw..

[2]  Enayat Allah Moallemi,et al.  A participatory exploratory modelling approach for long-term planning in energy transitions , 2017 .

[3]  Ronald Apriliyanto Halim,et al.  A scenario discovery study of the impact of uncertainties in the global container transport system on European ports , 2016 .

[4]  Warren E. Walker,et al.  Coping with the Wickedness of Public Policy Problems: Approaches for Decision Making under Deep Uncertainty , 2016 .

[5]  Emanuele Borgonovo,et al.  A new uncertainty importance measure , 2007, Reliab. Eng. Syst. Saf..

[6]  John D. Sterman,et al.  System Dynamics: Systems Thinking and Modeling for a Complex World , 2002 .

[7]  Jan H. Kwakkel,et al.  Using System Dynamics for Grand Challenges: The ESDMA Approach , 2015 .

[8]  Levent Yilmaz,et al.  Toward Agent-Supported and Agent-Monitored Model-Driven Simulation Engineering , 2015 .

[9]  Emanuele Borgonovo,et al.  Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help? , 2014, Risk analysis : an official publication of the Society for Risk Analysis.

[10]  Enayat Allah Moallemi,et al.  A dual narrative-modelling approach for evaluating socio-technical transitions in electricity sectors , 2017 .

[11]  Keith Beven,et al.  The future of distributed models: model calibration and uncertainty prediction. , 1992 .

[12]  Marko Hofmann,et al.  Simulation-based exploratory data generation and analysis (data farming): a critical reflection on its validity and methodology , 2013 .

[13]  Jim W. Hall,et al.  Sensitivity analysis of environmental models: A systematic review with practical workflow , 2014, Environ. Model. Softw..

[14]  Andrea Castelletti,et al.  Is robustness really robust? How different definitions of robustness impact decision-making under climate change , 2016, Climatic Change.

[15]  Sritika Chakladar A Model Driven Engineering Framework for Simulation Experiment Management , 2016 .

[16]  David G. Groves,et al.  A New Analytic Method for Finding Policy-Relevant Scenarios: , 2007 .

[17]  Joseph P. Romano,et al.  On the uniform asymptotic validity of subsampling and the bootstrap , 2012, 1204.2762.

[18]  Joseph R. Kasprzyk,et al.  Incorporating deeply uncertain factors into the many objective search process , 2017, Environ. Model. Softw..

[19]  Erik Pruyt,et al.  Scenario generation using adaptive sampling: The case of resource scarcity , 2016, Environ. Model. Softw..

[20]  Sibel Eker,et al.  A model-based analysis of biomethane production in the Netherlands and the effectiveness of the subsidization policy under uncertainty , 2015 .

[21]  Alice E. Smith,et al.  A model-driven engineering approach to simulation experiment design and execution , 2015, 2015 Winter Simulation Conference (WSC).

[22]  Jan H. Kwakkel,et al.  How Robust is a Robust Policy? Comparing Alternative Robustness Metrics for Robust Decision-Making , 2016 .

[23]  David G. Groves,et al.  A General, Analytic Method for Generating Robust Strategies and Narrative Scenarios , 2006, Manag. Sci..

[24]  E. A. Moallemi,et al.  Model-based multi-objective decision making under deep uncertainty from a multi-method design lens , 2018, Simul. Model. Pract. Theory.

[25]  Jack P. C. Kleijnen,et al.  State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments , 2005, INFORMS J. Comput..

[26]  Levent Yilmaz,et al.  The Goal-Hypothesis-Experiment framework: A generative cognitive domain architecture for simulation experiment management , 2016, 2016 Winter Simulation Conference (WSC).

[27]  Vincent Marchau,et al.  Classifying and communicating uncertainties in model-based policy analysis , 2010 .

[28]  Andrew Binley,et al.  GLUE: 20 years on , 2014 .

[29]  Robert J. Lempert,et al.  Computer-assisted reasoning , 2001, Comput. Sci. Eng..

[30]  Robert J Lempert,et al.  A new decision sciences for complex systems , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[31]  Joseph H. A. Guillaume,et al.  Fit for purpose? Building and evaluating a fast, integrated model for exploring water policy pathways , 2014, Environ. Model. Softw..

[32]  Harvey M. Wagner,et al.  Global Sensitivity Analysis , 1995, Oper. Res..

[33]  John H. Miller,et al.  Active Nonlinear Tests (Ants) of Complex Simulation Models , 1998 .

[34]  J. C. Helton,et al.  An Investigation of Uncertainty and Sensitivity Analysis Techniques for Computer Models , 1988 .

[35]  Mark E. Borsuk,et al.  Discovering plausible energy and economic futures under global change using multidimensional scenario discovery , 2013, Environ. Model. Softw..

[36]  Warren E. Walker,et al.  Developing dynamic adaptive policy pathways: a computer-assisted approach for developing adaptive strategies for a deeply uncertain world , 2015, Climatic Change.

[37]  Emanuele Borgonovo,et al.  Sensitivity analysis: A review of recent advances , 2016, Eur. J. Oper. Res..

[38]  Jing Wang,et al.  Parameter sensitivity analysis of crop growth models based on the extended Fourier Amplitude Sensitivity Test method , 2013, Environ. Model. Softw..

[39]  Micha Werner,et al.  Reduction of Monte-Carlo simulation runs for uncertainty estimation in hydrological modelling , 2003 .

[40]  Jan H. Kwakkel,et al.  Dynamic scenario discovery under deep uncertainty: The future of copper , 2013 .

[41]  Henrik Madsen,et al.  Uncertainty assessment of integrated distributed hydrological models using GLUE with Markov chain Monte Carlo sampling , 2006 .

[42]  Robert J. Lempert,et al.  Comparing Algorithms for Scenario Discovery , 2008 .

[43]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[44]  R. Lempert,et al.  Identifying and evaluating robust adaptive policy responses to climate change for water management agencies in the American west , 2010 .

[45]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[46]  R. Lempert,et al.  Shaping the Next One Hundred Years: New Methods for Quantitative Long-Term Policy Analysis , 2003 .

[47]  Sina Khatami,et al.  Water for Energy: Inconsistent Assessment Standards and Inability to Judge Properly , 2015 .

[48]  E. A. Moallemi,et al.  Narrative-informed exploratory analysis of energy transition pathways: A case study of India's electricity sector , 2017 .

[49]  Gudmund R. Iversen,et al.  Analysis of Variance , 2011, International Encyclopedia of Statistical Science.

[50]  Jan H. Kwakkel,et al.  Adaptive Robust Design under deep uncertainty , 2013 .

[51]  Warren E. Walker,et al.  Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty , 2013 .

[52]  Benjamin P. Bryant,et al.  Thinking Inside the Box , 2010 .

[53]  Adelinde M. Uhrmacher,et al.  SESSL , 2014, ACM Trans. Model. Comput. Simul..

[54]  A. Jakeman,et al.  Providing scientific certainty in predictive decision support: the role of closed questions , 2012 .

[55]  Andrea Castelletti,et al.  Robustness Metrics: How Are They Calculated, When Should They Be Used and Why Do They Give Different Results? , 2018 .

[56]  Vincent Marchau,et al.  Uncertainty in the Framework of Policy Analysis , 2013 .

[57]  Nicholas I. Fisher,et al.  Bump hunting in high-dimensional data , 1999, Stat. Comput..

[58]  Steve Bankes,et al.  Exploratory Modeling for Policy Analysis , 1993, Oper. Res..

[59]  Michael J. Ryan,et al.  Informing the Delineation of Input Uncertainty Space in Exploratory Modelling Using a Heuristic Approach , 2018 .

[60]  Ana Deletic,et al.  Many roads to Rome: The emergence of pathways from patterns of change through exploratory modelling of sustainability transitions , 2016, Environ. Model. Softw..

[61]  Joseph R. Kasprzyk,et al.  Many objective robust decision making for complex environmental systems undergoing change , 2012, Environ. Model. Softw..