What do I make of your Latinorum? Sensitivity auditing of mathematical modelling.

Sensitivity analysis, mandated by existing guidelines as a good practice to use in conjunction to mathematical modelling, is as such insufficient to ensure quality in the treatment of uncertainty of science for policy. If one accepts that policy-related science calls for an extension of the traditional internal, peer review-based methods of quality assurance to higher levels of supervision, where extended participation and explicit value judgments are necessary, then by the same token sensitivity analysis must extend beyond the technical exploration of the space of uncertain assumptions when the inference being sought via mathematical modelling is subject to relevant uncertainties and stakes. We thus provide seven rules to extend the use of sensitivity analysis (or how to apportion uncertainty in model-based inference among input factors) in a process of sensitivity auditing of models used in a policy context. Each rule will be illustrated by examples.

[1]  Leonard A. Smith,et al.  Policy: Clarify the limits of climate models , 2012, Nature.

[2]  J. Hansen,et al.  Perception of climate change , 2012, Proceedings of the National Academy of Sciences.

[3]  Mark Maslin,et al.  Uncertainty: Climate models at their limit? , 2012, Nature.

[4]  J. Hansen,et al.  Perception of climate change , 2012, Proceedings of the National Academy of Sciences.

[5]  D. Lindenmayer Merchants of Doubt. How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming , 2012 .

[6]  J. A. Wardekker,et al.  Assumptions in quantitative analyses of health risks of overhead power lines , 2012 .

[7]  Felix Salmon The formula that killed Wall Street , 2012 .

[8]  Jeroen P. van der Sluijs,et al.  On the contribution of external cost calculations to energy system governance: the case of a potential large-scale nuclear accident , 2011 .

[9]  Nick Barter,et al.  Freefall: Free Markets and the Sinking of the Global Economy , 2011 .

[10]  Joseph E. Stiglitz,et al.  RETHINKING MACROECONOMICS: WHAT FAILED, AND HOW TO REPAIR IT , 2011 .

[11]  D. Shahsavani,et al.  Variance-based sensitivity analysis of model outputs using surrogate models , 2011, Environ. Model. Softw..

[12]  Arthur C. Petersen,et al.  Post-Normal Science in Practice at the Netherlands Environmental Assessment Agency , 2011, Science, technology & human values.

[13]  Andrea Saltelli,et al.  Ratings and rankings: voodoo or science? , 2011, 1104.3009.

[14]  Arthur C. Petersen,et al.  A method for the analysis of assumptions in model-based environmental assessments , 2011, Environ. Model. Softw..

[15]  S. Lane,et al.  Doing flood risk science differently: an experiment in radical scientific method , 2011 .

[16]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[17]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[18]  Edward Leamer Tantalus on the Road to Asymptopia , 2010 .

[19]  Andrea Saltelli,et al.  Sensitivity analysis didn't help. A practitioner's critique of the Stern review , 2010 .

[20]  Marc Aerts,et al.  NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment , 2010 .

[21]  Paola Annoni,et al.  Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index , 2010, Comput. Phys. Commun..

[22]  Keith Beven,et al.  Uncertainty and good practice in hydrological prediction , 2010 .

[23]  Alexander M. Millkey The Black Swan: The Impact of the Highly Improbable , 2009 .

[24]  Michael Søgaard Jørgensen,et al.  Atmospheric composition change research: Time to go post-normal? , 2009 .

[25]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[26]  Jerome Ravetz Morals and manners in modern science , 2009, Nature.

[27]  Jerome R. Ravetz,et al.  Exploring the quality of evidence for complex and contested policy decisions , 2008 .

[28]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[29]  W. Nordhaus Critical Assumptions in the Stern Review on Climate Change , 2007, Science.

[30]  P. Boulanger Political uses of social indicators: overview and application to sustainable development indicators , 2007 .

[31]  V. Lakshmi Useless Arithmetic: Why Environmental Scientists Can't Predict the Future , 2007 .

[32]  Nassim Nicholas Taleb,et al.  The Black Swan: The Impact of the Highly Improbable , 2007 .

[33]  Funtowicz Silvio,et al.  Models of Science and Policy , 2007 .

[34]  More letters by Paul Feyerabend to Thomas S. Kuhn on Proto-Structure , 2006 .

[35]  W. Nordhaus The "Stern Review" on the Economics of Climate Change , 2006 .

[36]  J. P. van der Sluijs,et al.  Uncertainty, assumptions, and value commitments in the knowledge base of complex environmental problems , 2006 .

[37]  Jeroen P. van der Sluijs,et al.  Uncertainty as a monster in the science–policy interface: four coping strategies , 2005 .

[38]  G. Musser Flaw of Averages , 2005 .

[39]  A. Saltelli,et al.  Sensitivity analysis for chemical models. , 2005, Chemical reviews.

[40]  S. Funtowicz,et al.  Combining Quantitative and Qualitative Measures of Uncertainty in Model‐Based Environmental Assessment: The NUSAP System , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[41]  Stefano Tarantola,et al.  Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators , 2005 .

[42]  J. P. van der Sluijs,et al.  Uncertainty as a monster in the science-policy interface: four coping strategies. , 2005, Water science and technology : a journal of the International Association on Water Pollution Research.

[43]  Malte Meinshausen,et al.  On the Risk of Overshooting 2°C , 2005 .

[44]  David Michaels,et al.  Doubt is their product. , 2005, Scientific American.

[45]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[46]  Ângela Guimarães Pereira,et al.  ICT Tools to Support Public Participation in Water Resources Governance & Planning: Experiences from the Design and Testing of a Multi-Media Platform , 2003 .

[47]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[48]  Andrea Saltelli,et al.  Sensitivity Analysis for Importance Assessment , 2002, Risk analysis : an official publication of the Society for Risk Analysis.

[49]  Sandra Simonsen-Anderson,et al.  Safe and sound. , 2002, Nursing management.

[50]  OFFICE OF MANAGEMENT AND BUDGET Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies , 2002 .

[51]  Ângela Guimarães Pereira,et al.  From Technocratic to Participatory Decision Support Systems: Responding to the New Governance Initiatives , 2002 .

[52]  Houthuijs Djm,et al.  Magnetische velden van hoogspanningslijnen en leukemie bij kinderen , 2001 .

[53]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[54]  R. Pielke,et al.  Prediction : science, decision making, and the future of nature , 2000 .

[55]  Jean Baudrillard,et al.  Revenge of the crystal , 1999 .

[56]  Simon Shackley,et al.  Anchoring Devices in Science for Policy , 1998 .

[57]  Harvey M. Wagner,et al.  Global Sensitivity Analysis , 1995, Oper. Res..

[58]  Paul Hoyningen-Huene Two letters of Paul Feyerabend to Thomas S. Kühn on a draft of the structure of scientific revolutions , 1995 .

[59]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[60]  S. Funtowicz,et al.  Three types of risk assessment and the emergence of post-normal science , 1992 .

[61]  John D. Bredehoeft,et al.  Ground-water models cannot be validated , 1992 .

[62]  R. Rosen Life Itself: A Comprehensive Inquiry Into the Nature, Origin, and Fabrication of Life , 1991 .

[63]  S. Funtowicz,et al.  A New Scientific Methodology for Global Environmental Issues , 1991 .

[64]  S. Lwanga,et al.  Estimation and projection of adult AIDS cases: a simple epidemiological model. , 1991, Bulletin of the World Health Organization.

[65]  Jerome R. Ravetz,et al.  Uncertainty and Quality in Science for Policy , 1990 .

[66]  J. Baudrillard,et al.  Revenge of the Crystal: Selected Writings on the Modern Object and Its Destiny, 1968-1983 , 1990 .

[67]  Saunders Mac Lane IIASA and Modeling , 1988 .

[68]  Saunders Mac Lane IIASA's Credibility , 1988 .

[69]  Roger Schultz,et al.  Verification & validation , 1988 .

[70]  H. Brachinger,et al.  Decision analysis , 1997 .

[71]  Jerome R. Ravetz,et al.  Three Types of Risk Assessment: A Methodological Analysis , 1985 .

[72]  R. Michels The Return to Cosmology: Postmodern Science and the Theology of Nature , 1983 .

[73]  Stephen E. Toulmin,et al.  The Return of Cosmology: Postmodern Science and the Theology of Nature , 1983 .

[74]  Edward E. Leamer,et al.  Let's Take the Con Out of Econometrics , 1983 .

[75]  G. Hornberger,et al.  Approach to the preliminary analysis of environmental systems , 1981 .

[76]  Peter E. Kennedy,et al.  A Guide to Econometrics , 1980 .

[77]  R. Merton The Normative Structure of Science , 1973 .

[78]  I. Lakatos PROOFS AND REFUTATIONS (I)*† , 1963, The British Journal for the Philosophy of Science.

[79]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[80]  J. Geoghegan Return to Reason! , 1942, Nature.

[81]  F. Knight The economic nature of the firm: From Risk, Uncertainty, and Profit , 2009 .