Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models

PREFACE. 1. A WORKED EXAMPLE. 1.1 A simple model. 1.2 Modulus version of the simple model. 1.3 Six--factor version of the simple model. 1.4 The simple model 'by groups'. 1.5 The (less) simple correlated--input model. 1.6 Conclusions. 2. GLOBAL SENSITIVITY ANALYSIS FOR IMPORTANCE ASSESSMENT. 2.1 Examples at a glance. 2.2 What is sensitivity analysis? 2.3 Properties of an ideal sensitivity analysis method. 2.4 Defensible settings for sensitivity analysis. 2.5 Caveats. 3. TEST CASES. 3.1 The jumping man. Applying variance--based methods. 3.2 Handling the risk of a financial portfolio: the problem of hedging. Applying Monte Carlo filtering and variance--based methods. 3.3 A model of fish population dynamics. Applying the method of Morris. 3.4 The Level E model. Radionuclide migration in the geosphere. Applying variance--based methods and Monte Carlo filtering. 3.5 Two spheres. Applying variance based methods in estimation/calibration problems. 3.6 A chemical experiment. Applying variance based methods in estimation/calibration problems. 3.7 An analytical example. Applying the method of Morris. 4. THE SCREENING EXERCISE. 4.1 Introduction. 4.2 The method of Morris. 4.3 Implementing the method. 4.4 Putting the method to work: an analytical example. 4.5 Putting the method to work: sensitivity analysis of a fish population model. 4.6 Conclusions. 5. METHODS BASED ON DECOMPOSING THE VARIANCE OF THE OUTPUT. 5.1 The settings. 5.2 Factors Prioritisation Setting. 5.3 First--order effects and interactions. 5.4 Application of Si to Setting 'Factors Prioritisation'. 5.5 More on variance decompositions. 5.6 Factors Fixing (FF) Setting. 5.7 Variance Cutting (VC) Setting. 5.8 Properties of the variance based methods. 5.9 How to compute the sensitivity indices: the case of orthogonal input. 5.9.1 A digression on the Fourier Amplitude Sensitivity Test (FAST). 5.10 How to compute the sensitivity indices: the case of non--orthogonal input. 5.11 Putting the method to work: the Level E model. 5.11.1 Case of orthogonal input factors. 5.11.2 Case of correlated input factors. 5.12 Putting the method to work: the bungee jumping model. 5.13 Caveats. 6. SENSITIVITY ANALYSIS IN DIAGNOSTIC MODELLING: MONTE CARLO FILTERING AND REGIONALISED SENSITIVITY ANALYSIS, BAYESIAN UNCERTAINTY ESTIMATION AND GLOBAL SENSITIVITY ANALYSIS. 6.1 Model calibration and Factors Mapping Setting. 6.2 Monte Carlo filtering and regionalised sensitivity analysis. 6.2.1 Caveats. 6.3 Putting MC filtering and RSA to work: the problem of hedging a financial portfolio. 6.4 Putting MC filtering and RSA to work: the Level E test case. 6.5 Bayesian uncertainty estimation and global sensitivity analysis. 6.5.1 Bayesian uncertainty estimation. 6.5.2 The GLUE case. 6.5.3 Using global sensitivity analysis in the Bayesian uncertainty estimation. 6.5.4 Implementation of the method. 6.6 Putting Bayesian analysis and global SA to work: two spheres. 6.7 Putting Bayesian analysis and global SA to work: a chemical experiment. 6.7.1 Bayesian uncertainty analysis (GLUE case). 6.7.2 Global sensitivity analysis. 6.7.3 Correlation analysis. 6.7.4 Further analysis by varying temperature in the data set: fewer interactions in the model. 6.8 Caveats. 7. HOW TO USE SIMLAB. 7.1 Introduction. 7.2 How to obtain and install SIMLAB. 7.3 SIMLAB main panel. 7.4 Sample generation. 7.4.1 FAST. 7.4.2 Fixed sampling. 7.4.3 Latin hypercube sampling (LHS). 7.4.4 The method of Morris. 7.4.5 Quasi--Random LpTau. 7.4.6 Random. 7.4.7 Replicated Latin Hypercube (r--LHS). 7.4.8 The method of Sobol'. 7.4.9 How to induce dependencies in the input factors. 7.5 How to execute models. 7.6 Sensitivity analysis. 8. FAMOUS QUOTES: SENSITIVITY ANALYSIS IN THE SCIENTIFIC DISCOURSE. REFERENCES. INDEX.

[1]  Stefano Tarantola,et al.  Sensitivity analysis in model calibration: GSA-GLUE approach , 2001 .

[2]  M. Stein.  Large sample properties of simulations using latin hypercube sampling , 1987 .

[3]  Saltelli Andrea,et al.  Statistical Techniques and Participatory Approaches for the Compilation of the European Internal Market Index 1992-2001 , 2002 .

[4]  J. Seinfeld,et al.  Automatic sensitivity analysis of kinetic mechanisms , 1979 .

[5]  J. S. Hunter,et al.  Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. , 1979 .

[6]  N. Draper,et al.  Applied Regression Analysis. , 1967 .

[7]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[8]  S. Tarantola,et al.  State-of-the-art Report on Current Methodologies and Practices for Composite Indicator Development , 2002 .

[9]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[10]  A. Saltelli,et al.  Sensitivity analysis of an environmental model: an application of different analysis methods , 1997 .

[11]  R. Iman,et al.  A distribution-free approach to inducing rank correlation among input variables , 1982 .

[12]  Keith Beven,et al.  The use of generalised likelihood measures for uncertainty estimation in high order models of environmental systems , 2000 .

[13]  A. Saltelli,et al.  Tackling quantitatively large dimensionality problems , 1999 .

[14]  A. Saltelli,et al.  Sensitivity Anaysis as an Ingredient of Modeling , 2000 .

[15]  I. Sobol,et al.  About the use of rank transformation in sensitivity analysis of model output , 1995 .

[16]  Brian D. Ripley,et al.  Stochastic Simulation , 2005 .

[17]  H. Caswell.  Matrix population models : construction, analysis, and interpretation , 2001 .

[18]  M. B. Beck,et al.  Identification of model structure for aquatic ecosystems using regionalized sensitivity analysis. , 2001, Water science and technology : a journal of the International Association on Water Pollution Research.

[19]  Kristin Shrader-Frechette,et al.  The Precautionary Principle: Scientific Uncertainty and Type I and Type II Errors , 1997 .

[20]  Bernardus Wilhelmus Maria Bettonvil,et al.  Detection of important factors by sequential bifurcation , 1990 .

[21]  A. Saltelli,et al.  Sensitivity analysis: Could better methods be used? , 1999 .

[22]  Mario Paruggia,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2006 .

[23]  Riccardo Rebonato,et al.  Interest-Rate Option Models: Understanding, Analysing and Using Models for Exotic Interest-Rate Options , 1998 .

[24]  Andrew Stirling,et al.  Valuing the environmental impacts of electricity production: a critical review of some "first generation" studies , 1998 .

[25]  Ronald L. Iman,et al.  A FORTRAN-77 PROGRAM AND USER'S GUIDE FOR THE GENERATION OF LATIN HYPERCUBE AND RANDOM SAMPLES FOR USE WITH COMPUTER MODELS , 1984 .

[26]  Nenad Karajić,et al.  ENVIRONMENTAL POLICY PERFORMANCE INDICATORS (Albert Adriaanse) , 1995 .

[27]  A. Saltelli,et al.  Importance measures in global sensitivity analysis of nonlinear models , 1996 .

[28]  T. Turányi.  Sensitivity analysis of complex kinetic systems. Tools and applications , 1990 .

[29]  Keith Beven,et al.  The future of distributed models: model calibration and uncertainty prediction. , 1992 .

[30]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[31]  K. Shuler,et al.  Nonlinear sensitivity analysis of multiparameter model systems , 1977 .

[32]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[33]  John Geweke,et al.  Federal Reserve Bank of Minneapolis Research Department Staff Report 249 Using Simulation Methods for Bayesian Econometric Models: Inference, Development, and Communication , 2022 .

[34]  Peter C. Young,et al.  Data-based mechanistic modelling, generalised sensitivity and dominant mode analysis , 1999 .

[35]  K. Beven,et al.  Bayesian Estimation of Uncertainty in Runoff Prediction and the Value of Data: An Application of the GLUE Approach , 1996 .

[36]  Nong Shang,et al.  Parameter uncertainty and interaction in complex environmental models , 1994 .

[37]  Stefano Tarantola,et al.  Uncertainty and sensitivity analysis: tools for GIS-based model implementation , 2001, Int. J. Geogr. Inf. Sci..

[38]  T. Ishigami,et al.  An importance quantification technique in uncertainty analysis for computer models , 1990, [1990] Proceedings. First International Symposium on Uncertainty Modeling and Analysis.

[39]  S. C. Cotter.  A screening design for factorial experiments with interactions , 1979 .

[40]  Jack P. C. Kleijnen,et al.  Searching for important factors in simulation models with many factors: Sequential bifurcation , 1997 .

[41]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[42]  Ilya M. Sobol,et al.  Sensitivity Estimates for Nonlinear Mathematical Models , 1993 .

[43]  Michael D. McKay,et al.  Evaluating Prediction Uncertainty , 1995 .

[44]  Keith Beven,et al.  Dynamic real-time prediction of flood inundation probabilities , 1998 .

[45]  P. Young,et al.  Simplicity out of complexity in environmental modelling: Occam's razor revisited. , 1996 .

[46]  Andrew Stirling.  On Science and Precaution in the Management of Technological Risk: Volume II -case studies , 1999 .

[47]  Edward E. Leamer,et al.  Let's Take the Con Out of Econometrics , 1983 .

[48]  H Rabitz,et al.  Systems Analysis at the Molecular Scale , 1989, Science.

[49]  Robert C. Spear,et al.  Large simulation models: calibration, uniqueness and goodness of fit , 1997 .

[50]  A. Stirling.  On science and precaution in the management of technological risk: volume I - a synthesis report of case studies , 1999 .

[51]  H. Rabitz,et al.  Efficient input-output model representations , 1999 .

[52]  Andrea Saltelli,et al.  Radionuclide migration in the geosphere : a 1D advective and dispersive transport module for use in probabilistics system assessment codes , 1991 .

[53]  M. Jansen,et al.  Monte Carlo estimation of uncertainty contributions from several independent multivariate sources. , 1994 .

[54]  Jon C. Helton,et al.  Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal , 1993 .

[55]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[56]  John Crank,et al.  The Mathematics Of Diffusion , 1956 .

[57]  Olufemi O. Osidele.  Reachable Futures, Structural Change, and the Practical Credibility of Environmental Simulation Models , 2002 .

[58]  Renata J. Romanowicz,et al.  Bayesian uncertainty estimation methodology applied to air pollution modelling , 2000 .

[59]  G. Hornberger,et al.  Approach to the preliminary analysis of environmental systems , 1981 .

[60]  Yonathan Bard,et al.  Nonlinear parameter estimation , 1974 .

[61]  A. Saltelli,et al.  On the Relative Importance of Input Factors in Mathematical Models , 2002 .

[62]  A. Saltelli,et al.  Making best use of model evaluations to compute sensitivity indices , 2002 .

[63]  B. Fischhoff,et al.  Assessing uncertainty in physical constants , 1986 .