Evaluating integrated assessment models of global climate change

Integrated Assessment Models of global climate change (IAMs) are an established tool to study interlinkages between the human and the natural system. Insights from these complex models are widely used to advise policy-makers and to inform the general public. But up to now there has been little understanding of how these models can be evaluated and community-wide standards are missing. To answer this urgent question is a challenge because the systems are open and their future behavior is fundamentally unknown. In this paper, we discuss ways to overcome these problems. Reflecting on experience from other modeling communities, we develop an evaluation framework for IAM of global climate change. It builds on a systematic and transparent step-by-step demonstration of a model's usefulness testing the plausibility of its behavior. Steps in the evaluation hierarchy are: setting up an evaluation framework, evaluation of the conceptual model, code verification and documentation, model evaluation, uncertainty and sensitivity analysis, documentation of the evaluation process, and communication with stakeholders. An important element in evaluating IAM of global climate change is the use of stylized behavior patterns derived from historical observation. The discussion of two examples is offered in this paper.

[1]  Danièle Revel,et al.  IPCC Special Report on Renewable Energy Sources and Climate Change Mitigation , 2011 .

[2]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[3]  S. Funtowicz,et al.  Combining Quantitative and Qualitative Measures of Uncertainty in Model‐Based Environmental Assessment: The NUSAP System , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[4]  François Bousquet,et al.  Modelling with stakeholders , 2010, Environ. Model. Softw..

[5]  M. Buchanan It's a (stylized) fact! , 2012 .

[6]  Richard S. J. Tol,et al.  The Stern Review of the Economics of Climate Change: A Comment , 2006 .

[7]  C. I. Jones,et al.  The New Kaldor Facts: Ideas, Institutions, Population, and Human Capital , 2009 .

[8]  David W. Cash,et al.  Knowledge systems for sustainable development , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[9]  J. P. van der Sluijs,et al.  A checklist for quality assistance in environmental modelling , 2006 .

[10]  Yaman Barlas,et al.  Formal aspects of model validity and validation in system dynamics , 1996 .

[11]  Yuan Guo-xing,et al.  Verification and Validation in Scientific Computing Code , 2010 .

[12]  Jordan Macknick,et al.  Energy and CO2 emission data uncertainties , 2011 .

[13]  John P. Weyant,et al.  A perspective on integrated assessment , 2009 .

[14]  Edward J. Rykiel,et al.  Testing ecological models: the meaning of validation , 1996 .

[15]  William D. Nordhaus,et al.  A Review of the Stern Review on the Economics of Climate Change , 2007 .

[16]  M. Thring World Energy Outlook , 1977 .

[17]  Marcus S. Fisher Software verification and validation - an engineering and scientific approach , 2006 .

[18]  A. Grubler,et al.  Future capacity growth of energy technologies: are scenarios consistent with historical evidence? , 2013, Climatic Change.

[19]  M. B. Beck,et al.  On the problem of model validation for predictive exposure assessments , 1997 .

[20]  Robert G. Sargent,et al.  Verification and validation: verification and validation of simulation models , 2003, WSC '03.

[21]  I. Sobol Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[22]  Aie World Energy Outlook 2011 , 2011 .

[23]  N. Kaldor Capital Accumulation and Economic Growth , 1961 .

[24]  Anthony J. Jakeman,et al.  Progress in integrated assessment and modelling , 2002, Environ. Model. Softw..

[25]  Patricia Gober,et al.  Credibility, salience, and legitimacy of boundary objects: water managers' assessment of a simulation model in an immersive decision theater , 2010 .

[26]  Jerome R. Ravetz,et al.  Uncertainty and Quality in Science for Policy , 1990 .

[27]  Yaman Barlas,et al.  Philosophical roots of model validation: Two paradigms , 1990 .

[28]  Andrea Saltelli,et al.  Sensitivity analysis didn't help. A practitioner's critique of the Stern review , 2010 .

[29]  Vaclav Smil,et al.  Perils of Long-Range Energy Forecasting , 2000 .

[30]  Luis Samaniego,et al.  Technical assessment and evaluation of environmental models and software: Letter to the Editor , 2011, Environ. Model. Softw..

[31]  Yaman Barlas,et al.  Model simplification and validation with indirect structure validity tests , 2006 .

[32]  Paul R. Brewer,et al.  Whose Science Do You Believe? Explaining Trust in Sources of Scientific Information About the Environment , 2013 .

[33]  N. Stern The Economics of Climate Change: Implications of Climate Change for Development , 2007 .

[34]  Anand Patwardhan,et al.  Assessing integrated assessments , 1996 .

[35]  W. Nordhaus The "Stern Review" on the Economics of Climate Change , 2006 .

[36]  Richard M. Burton,et al.  Computational Laboratories for Organization Science: Questions, Validity and Docking , 2003, Comput. Math. Organ. Theory.

[37]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[38]  Laurence Smith,et al.  The role of expert opinion in environmental modelling , 2012, Environ. Model. Softw..

[39]  Jeroen P. van der Sluijs,et al.  A way out of the credibility crisis of models used in integrated environmental assessment , 2002 .

[40]  K. Beven Towards a coherent philosophy for modelling the environment , 2002, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[41]  K. Taylor,et al.  An overview of results from the Coupled Model Intercomparison Project , 2003 .

[42]  Arthur C. Petersen,et al.  A method for the analysis of assumptions in model-based environmental assessments , 2011, Environ. Model. Softw..

[43]  Stephen H. Schneider,et al.  Integrated assessment modeling of global climate change: Transparent rational tool for policy making or opaque screen hiding value‐laden assumptions? , 1997 .

[44]  John D. Bredehoeft,et al.  Ground-water models cannot be validated , 1992 .

[45]  David V. Budescu,et al.  Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change , 2009, Psychological science.

[46]  Muhittin Oral,et al.  In search of a valid view of model validation for operations research , 1993 .

[47]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[48]  Arnulf Grubler,et al.  Transitions in Energy Use , 2004 .

[49]  John P. Weyant,et al.  Modeling for insights, not numbers: the experiences of the energy modeling forum , 1982 .

[50]  Global Energy Assessment Writing Team Global Energy Assessment: Toward a Sustainable Future , 2012 .

[51]  Y. Kaya Impact of carbon dioxide emission control on GNP growth : Interpretation of proposed scenarios , 1989 .

[52]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[53]  Alexander Rosenberg,et al.  Instrumental Biology, Or, The Disunity Of Science , 1994 .

[54]  Sheila Jasanoff,et al.  Testing Time for Climate Science , 2010, Science.

[55]  Mark Jaccard,et al.  Hybrid Modeling: New Answers to Old Challenges Introduction to the Special Issue of The Energy Journal , 2006 .

[56]  K. Beven Reply to ‘The emergence of a new kind of relativism in environmental modelling: a commentary’ by Philippe Baveye , 2004, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[57]  Michael S. Carolan,et al.  The Bright- and Blind-Spots of Science: Why Objective Knowledge is not Enough to Resolve Environmental Controversies , 2008 .

[58]  Richard G. Oderwald,et al.  Corroborating models with model properties , 1993 .

[59]  Ortwin Renn,et al.  Credibility and trust in risk communication , 1991 .

[60]  Valentina Bosetti,et al.  The Future Prospect of PV and CSP Solar Technologies: An Expert Elicitation Survey , 2012 .

[61]  M. Weitzman A Review of The Stern Review on the Economics of Climate Change , 2007 .