A checklist for model credibility, salience, and legitimacy to improve information transfer in environmental policy assessments

Modelers involved in environmental policy assessments are commonly confronted with the lack of uptake of model output by policy actors. Actors have different expectations of models, condensed into three quality criteria: credibility, salience, and legitimacy. The fulfilment of quality criteria is also dynamic as expectations vary, change, and possibly counteract each other. We present a checklist for modelers involved in model-based assessments that is aimed at the identification and monitoring of issues, limitations and trade-offs regarding model quality criteria. It draws upon the literature of integrated assessments as well as case study analysis of environmental policy assessments for the Dutch government, based on expert interviews and embedded experience. The checklist is intended to be consulted during assessments; its application may result in greater awareness among modelers involved in assessments regarding model quality criteria, and may positively affect the uptake of model-based knowledge from environmental policy assessments by policy actors. Model credibility, salience and legitimacy are affected by various factors.Factors are summarized based on interviews, literature and a case study.A model evaluation checklist for modelers is presented to help detect such factors.The checklist may improve the uptake of model-based output by decision makers.

[1]  Ray J. Paul,et al.  On simulation model complexity , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[2]  Marc Schut,et al.  Beyond the research-policy interface. Boundary arrangements at research-stakeholder interfaces in the policy debate on biofuel sustainability in Mozambique , 2013 .

[3]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[4]  Dan Cornford,et al.  Managing uncertainty in integrated environmental modelling: The UncertWeb framework , 2013, Environ. Model. Softw..

[5]  Arnim Wiek,et al.  The evolution of the IPCC's emissions scenarios , 2009 .

[6]  Laurence Smith,et al.  The role of expert opinion in environmental modelling , 2012, Environ. Model. Softw..

[7]  David W. Cash,et al.  Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making , 2002 .

[8]  Ângela Guimarães Pereira,et al.  RIVM/MNP Guidance for Uncertainty Assessment and Communication: Detailed Guidance , 2003 .

[9]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[10]  Arthur C. Petersen,et al.  The reflective futures practitioner: Balancing salience, credibility and legitimacy in generating foresight knowledge with stakeholders , 2015 .

[11]  Jens Christian Refsgaard,et al.  Modelling guidelinesterminology and guiding principles , 2004 .

[12]  M. Hare,et al.  Exploring the Gap Between Water Managers and Researchers: Difficulties of Model-Based Tools to Support Practical Water Management , 2007 .

[13]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[14]  Claire Waterton,et al.  Rethinking biodiversity: from goods and services to “living with” , 2013 .

[15]  Yuqiong Liu,et al.  Linking science with environmental decision making: Experiences from an integrated modeling approach to supporting sustainable water resources management , 2008, Environ. Model. Softw..

[16]  Arthur C. Petersen,et al.  A method for the analysis of assumptions in model-based environmental assessments , 2011, Environ. Model. Softw..

[17]  Pebesma Edzer,et al.  Managing uncertainty in integrated environmental modelling frameworks: The UncertWeb framework , 2012 .

[18]  Holger Meinke,et al.  Actionable climate knowledge: from analysis to synthesis , 2006 .

[19]  Thomas Jahn,et al.  An actor-specific guideline for quality assurance in transdisciplinary research , 2015 .

[20]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[21]  Thomas Bauler,et al.  An analytical framework to discuss the usability of (environmental) indicators for policy , 2012 .

[22]  Anthony J. Jakeman,et al.  Selecting among five common modelling approaches for integrated environmental assessment and management , 2013, Environ. Model. Softw..

[23]  Rebecca A. Kelly,et al.  A formal framework for scenario development in support of environmental decision-making , 2009, Environ. Model. Softw..

[24]  Bent Flyvbjerg,et al.  Social science that matters , 2006 .

[25]  Luis Samaniego,et al.  Technical assessment and evaluation of environmental models and software: Letter to the Editor , 2011, Environ. Model. Softw..

[26]  R. Hoppe,et al.  Cultures of Public Policy Problems , 2002 .

[27]  Peter A. Vanrolleghem,et al.  Uncertainty in the environmental modelling process - A framework and guidance , 2007, Environ. Model. Softw..

[28]  Edward J. Rykiel,et al.  Testing ecological models: the meaning of validation , 1996 .

[29]  Axel Volkery,et al.  Your Vision or My Model? Lessons from Participatory Land Use Scenario Development on a European Scale , 2008 .

[30]  E. McNie Reconciling the supply of scientific information with user demands: an analysis of the problem and review of the literature , 2007 .

[31]  Thomas Jahn,et al.  Transdisciplinarity in the Practice of Research , 2008 .

[32]  Soroosh Sorooshian,et al.  A framework for development and application of hydrological models , 2001, Hydrology and Earth System Sciences.

[33]  François Bousquet,et al.  Modelling with stakeholders , 2010, Environ. Model. Softw..

[34]  Guillermo P Podesta,et al.  Interdisciplinary production of knowledge with participation of stakeholders: A case study of a collaborative project on climate variability, human decisions and agricultural ecosystems in the Argentine Pampas , 2013 .

[35]  Volker Grimm,et al.  Ecological models supporting environmental decision making: a strategy for the future. , 2010, Trends in ecology & evolution.

[36]  Warren E. Walker,et al.  Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support , 2003 .

[37]  Patricia Gober,et al.  Credibility, salience, and legitimacy of boundary objects: water managers' assessment of a simulation model in an immersive decision theater , 2010 .

[38]  Peter A. Tamás,et al.  Should I take this seriously? A simple checklist for calling bullshit on policy supporting research , 2014 .

[39]  Laxman Joshi,et al.  Users’ perspectives on validity of a simulation model for natural resource management , 2011 .

[40]  Jennie C. Stephens,et al.  Diverse Perceptions of Stakeholder Engagement within an Environmental Modeling Research Team , 2013, Journal of Environmental Studies and Sciences.

[41]  G. Finnveden,et al.  Scenario types and techniques: Towards a user's guide , 2006 .

[42]  Simo Sarkki,et al.  Adding ‘iterativity’ to the credibility, relevance, legitimacy: A novel scheme to highlight dynamic aspects of science–policy interfaces , 2015 .

[43]  Suzanne A. Pierce,et al.  Modelling with stakeholders e Next generation , 2015 .

[44]  Christopher J. Lord,et al.  Democracy in the European Union , 1998 .

[45]  Martin K. van Ittersum,et al.  How, when, and for what reasons does land use modelling contribute to societal problem solving? , 2011, Environ. Model. Softw..

[46]  Anthony J. Jakeman,et al.  Environmental decision support systems (EDSS) development - Challenges and best practices , 2011, Environ. Model. Softw..

[47]  Matthias Bergmann,et al.  Methods for Transdisciplinary Research: A Primer for Practice , 2013 .

[48]  Clare Ginger,et al.  Integrating knowledge, interests and values through modelling in participatory processes: dimensions of legitimacy , 2014 .

[49]  Pim Martens,et al.  Transdisciplinary research in sustainability science: practice, principles, and challenges , 2012, Sustainability Science.

[50]  Christine J. Kirchhoff,et al.  Narrowing the climate information usability gap , 2012 .

[51]  Erik Skärbäck,et al.  Solving Problems in Social–Ecological Systems: Definition, Practice and Barriers of Transdisciplinary Research , 2013, AMBIO.