Sensitivity analysis of model output: an investigation of new techniques

Sensitivity Analysis (SA) of model output investigates the relationship between the predictions of a model, possibly implemented in a computer program, and its input parameters. Such an analysis is relevant for a number of practices, including quality assurance of models and codes, and the identification of crucial regions in the parameter space. This note compares established techniques with variants, such as a modified version of the Hora and Iman importance measure (SANDIA Laboratory Report SAND85-2839, 19891, or new methods, such as the iterated fractional factorial design (Andres, Hajas, Report in prep. for AECL, Pinawa, Canada, 1991). Comparison is made on the basis of method reproducibility and of method accuracy. The former is a measure of how well SA predictions are replicated when repeating the analysis on different samples taken from the same input parameters space. The latter deals with the physical correctness of the SA results. The present article is a sequel to an earlier study in this journal (Saltelli, Homma, Comp. Stat. and Data Anal. 13 (1) 1992, 73-94 of limitations in existing SA techniques, where the inadequacy of existing schemes to deal with non-monotonic relationships within the model was pointed out. International benchmark test models were taken from the field of performance analysis of nuclear waste disposal in geological formations. The results based on these models show that the modified version of the Hora and Iman method proposed in this paper is extremely robust, when compared with the other existing statistics, even in the presence of model non-monotonicities. This importance measure is accurate, although its application is demanding - in terms of computer time - for system with very large numbers of input parameters. Its predictions are also among the most reproducible. The newly proposed iterated fractional factorial design appears to score the best in reproducibility. The accuracy of this latter method demands further investigation.

[1]  R. Iman,et al.  Rank correlation plots for use with correlated input variables , 1982 .

[2]  Saltelli Andrea,et al.  Safety Assessment for Nuclear Waste Disposal. Some Observations about Actual Risk Calculations , 1988 .

[3]  A. Saltelli,et al.  Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques , 1990 .

[4]  Gerald V. Poje.  Resources Needed for New Risk Analysis Opportunities1 , 1988 .

[5]  Andrea Saltelli,et al.  Sensitivity analysis for model output: performance of black box techniques on three international benchmark exercises , 1992 .

[6]  Andrea Saltelli,et al.  Techniques for Uncertainty and Sensitivity Analyses , 1989 .

[7]  Ronald L. Iman,et al.  Comparison of Maximus/Bounding and Bayes/Monte Carlo for fault tree uncertainty analysis , 1986 .

[8]  J. C. Helton,et al.  A COMPARISON OF UNCERTAINTY AND SENSITIVITY ANALYSIS TECHNIQUES FOR COMPUTER MODELS , 1985 .

[9]  Saltelli Andrea,et al.  Performances of Non-Parametric Statistics in Sensitivity Analysis and Parameter Ranking , 1988 .

[10]  Andrea Saltelli,et al.  The Role of the Code Intercomparison Exercises: Activities of The Probabilistic System Assessment Codes Group , 1989 .

[11]  Jon C. Helton,et al.  An Approach to Sensitivity Analysis of Computer Models: Part II - Ranking of Input Variables, Response Surface Validation, Distribution Effect and Technique Synopsis , 1981 .

[12]  R. Iman,et al.  The Use of the Rank Transform in Regression , 1979 .

[13]  T. Ishigami,et al.  An importance quantification technique in uncertainty analysis for computer models , 1990, [1990] Proceedings. First International Symposium on Uncertainty Modeling and Analysis.

[14]  D. Whitefield,et al.  A review of: “Practical Nonpararnetric Statistics. By W. J. CONOVER. (New York: Wiley, 1971.) [Pl" x+462.] £5·25. , 1972 .

[15]  S. G. Carlyle,et al.  PSACOIN level 0 intercomparison-an international verification exercise on a hypothetical safety assessment case study , 1989, [1989] Proceedings of the Twenty-Second Annual Hawaii International Conference on System Sciences. Volume II: Software Track.

[16]  Jay D. Johnson,et al.  Uncertainty and sensitivity analysis of a dry containment test problem for the MAEROS aerosol model , 1989 .

[17]  James E. Campbell,et al.  An Approach to Sensitivity Analysis of Computer Models: Part I—Introduction, Input Variable Selection and Preliminary Variable Assessment , 1981 .