What do I make of your Latinorum? Sensitivity auditing of mathematical modelling.

Sensitivity analysis, mandated by existing guidelines as a good practice to use in conjunction to mathematical modelling, is as such insufficient to ensure quality in the treatment of uncertainty of science for policy. If one accepts that policy-related science calls for an extension of the traditional internal, peer review-based methods of quality assurance to higher levels of supervision, where extended participation and explicit value judgments are necessary, then by the same token sensitivity analysis must extend beyond the technical exploration of the space of uncertain assumptions when the inference being sought via mathematical modelling is subject to relevant uncertainties and stakes. We thus provide seven rules to extend the use of sensitivity analysis (or how to apportion uncertainty in model-based inference among input factors) in a process of sensitivity auditing of models used in a policy context. Each rule will be illustrated by examples.

[1]  C. Macilwain Safe and sound? , 2006, Nature.

[2]  S. Funtowicz,et al.  Combining Quantitative and Qualitative Measures of Uncertainty in Model‐Based Environmental Assessment: The NUSAP System , 2005, Risk analysis : an official publication of the Society for Risk Analysis.

[3]  Sam Savage,et al.  The Flaw of Averages , 2002 .

[4]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[5]  P. Boulanger Political uses of social indicators: overview and application to sustainable development indicators , 2007 .

[6]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[7]  S. Funtowicz,et al.  A New Scientific Methodology for Global Environmental Issues , 1991 .

[8]  S. Lane,et al.  Doing flood risk science differently: an experiment in radical scientific method , 2011 .

[9]  J. A. Wardekker,et al.  Assumptions in quantitative analyses of health risks of overhead power lines , 2012 .

[10]  J. Hansen,et al.  Perception of climate change , 2012, Proceedings of the National Academy of Sciences.

[11]  Marc Aerts,et al.  NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment , 2010 .

[12]  Harvey M. Wagner,et al.  Global Sensitivity Analysis , 1995, Oper. Res..

[13]  Jerome R. Ravetz,et al.  Exploring the quality of evidence for complex and contested policy decisions , 2008 .

[14]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[15]  R. Merton The Normative Structure of Science , 1973 .

[16]  G. Hornberger,et al.  Approach to the preliminary analysis of environmental systems , 1981 .

[17]  F. Knight The economic nature of the firm: From Risk, Uncertainty, and Profit , 2009 .

[18]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[19]  W. Nordhaus The "Stern Review" on the Economics of Climate Change , 2006 .

[20]  Peter E. Kennedy A Guide to Econometrics , 1979 .

[21]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[22]  Funtowicz Silvio,et al.  Models of Science and Policy , 2007 .

[23]  Nick Barter,et al.  Freefall: Free Markets and the Sinking of the Global Economy , 2011 .

[24]  Edward Leamer Tantalus on the Road to Asymptopia , 2010 .

[25]  Andrea Saltelli,et al.  Ratings and rankings: voodoo or science? , 2011, 1104.3009.

[26]  Jerome R. Ravetz,et al.  Uncertainty and Quality in Science for Policy , 1990 .

[27]  Houthuijs Djm,et al.  Magnetische velden van hoogspanningslijnen en leukemie bij kinderen , 2001 .

[28]  Paola Annoni,et al.  Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index , 2010, Comput. Phys. Commun..

[29]  Keith Beven,et al.  Uncertainty and good practice in hydrological prediction , 2010 .

[30]  Andrea Saltelli,et al.  Sensitivity Analysis for Importance Assessment , 2002, Risk analysis : an official publication of the Society for Risk Analysis.

[31]  S. Toulmin Return to Reason , 2001 .

[32]  Simon Shackley,et al.  Anchoring Devices in Science for Policy , 1998 .

[33]  Stefano Tarantola,et al.  Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators , 2005 .

[34]  Mark Maslin,et al.  Uncertainty: Climate models at their limit? , 2012, Nature.

[35]  Jeroen P. van der Sluijs,et al.  On the contribution of external cost calculations to energy system governance: the case of a potential large-scale nuclear accident , 2011 .

[36]  David Michaels,et al.  Doubt is their product. , 2005, Scientific American.

[37]  Edward E. Leamer,et al.  Let's Take the Con Out of Econometrics , 1983 .

[38]  Arthur C. Petersen,et al.  A method for the analysis of assumptions in model-based environmental assessments , 2011, Environ. Model. Softw..

[39]  Joseph E. Stiglitz,et al.  RETHINKING MACROECONOMICS: WHAT FAILED, AND HOW TO REPAIR IT , 2011 .

[40]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[41]  Jerome Ravetz Morals and manners in modern science , 2009, Nature.

[42]  John D. Bredehoeft,et al.  Ground-water models cannot be validated , 1992 .

[43]  D. Lindenmayer Merchants of Doubt. How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming , 2012 .

[44]  S. Lwanga,et al.  Estimation and projection of adult AIDS cases: a simple epidemiological model. , 1991, Bulletin of the World Health Organization.

[45]  Ângela Guimarães Pereira,et al.  ICT Tools to Support Public Participation in Water Resources Governance & Planning: Experiences from the Design and Testing of a Multi-Media Platform , 2003 .

[46]  Saunders Mac Lane IIASA's Credibility , 1988 .

[47]  Ângela Guimarães Pereira,et al.  From Technocratic to Participatory Decision Support Systems: Responding to the New Governance Initiatives , 2002 .

[48]  W. Nordhaus Critical Assumptions in the Stern Review on Climate Change , 2007, Science.

[49]  Nassim Nicholas Taleb,et al.  The Black Swan: The Impact of the Highly Improbable , 2007 .

[50]  S. Funtowicz,et al.  Three types of risk assessment and the emergence of post-normal science , 1992 .

[51]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[52]  Malte Meinshausen,et al.  On the Risk of Overshooting 2°C , 2005 .

[53]  A. Saltelli,et al.  Sensitivity analysis for chemical models. , 2005, Chemical reviews.

[54]  J. Stiglitz Freefall: Free Markets and the Sinking of the Global Economy , 2010 .

[55]  J. P. van der Sluijs,et al.  Uncertainty as a monster in the science-policy interface: four coping strategies. , 2005, Water science and technology : a journal of the International Association on Water Pollution Research.

[56]  Leonard A. Smith,et al.  Policy: Clarify the limits of climate models , 2012, Nature.

[57]  Andrea Saltelli,et al.  Sensitivity analysis didn't help. A practitioner's critique of the Stern review , 2010 .

[58]  R. Michels The Return to Cosmology: Postmodern Science and the Theology of Nature , 1983 .

[59]  Arthur C. Petersen,et al.  Post-Normal Science in Practice at the Netherlands Environmental Assessment Agency , 2011, Science, technology & human values.

[60]  Saunders Mac Lane IIASA and Modeling , 1988 .

[61]  S. Funtowicz,et al.  Science for the Post-Normal Age , 1993, Commonplace.

[62]  Stephen E. Toulmin,et al.  The Return of Cosmology: Postmodern Science and the Theology of Nature , 1983 .

[63]  Felix Salmon The formula that killed Wall Street , 2012 .

[64]  Michael Søgaard Jørgensen,et al.  Atmospheric composition change research: Time to go post-normal? , 2009 .

[65]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[66]  Jean Baudrillard,et al.  Revenge of the crystal , 1999 .

[67]  Roger Schultz,et al.  Verification & validation , 1988 .

[68]  I. Lakatos PROOFS AND REFUTATIONS (I)*† , 1963, The British Journal for the Philosophy of Science.