Sensitivity analysis for model output: performance of black box techniques on three international benchmark exercises

Abstract The paper analyses the difficulties of performing sensitivity analysis on the output of complex models. To this purpose a number of selected non-parametric statistics techniques are applied to model outputs without assuming knowledge of the model structure, ie as to a black box. The techniques employed are mainly concerned with the analysis of the rank transformation of both input and output variables (eg standardised rank regression coefficients, model coefficient of determination on ranks...). The test models taken into consideration are three benchmarks of the Probabilistic System Assessment Code (PSAC) User Group, an international working party coordinated by the OECD/NEA. They describe nuclide chain transport through a multi-barrier system (near field, geosphere, biosphere) and are employed in the analysis of the safety of a nuclear waste disposal in a geological formation. Due to the large uncertainties affecting the system these models are normally run within a Monte Carlo driver in order to characterise the distribution of the model output. A crucial step in the analysis of the system is the study of the sensitivity of the model output to the value of its input parameters. This study may be complicated by factors such as the complexity of the model, its non-linearity and non-monotonicity and others. The problem is discussed with reference to the three test cases and model non-monotonicity is shown to be particularly difficult to handle with the employed techniques. Alternative approaches to sensitivity analysis are also touched upon.

[1]  S. G. Carlyle,et al.  PSACOIN level 0 intercomparison-an international verification exercise on a hypothetical safety assessment case study , 1989, [1989] Proceedings of the Twenty-Second Annual Hawaii International Conference on System Sciences. Volume II: Software Track.

[2]  Ronald L. Iman,et al.  FORTRAN 77 program and user's guide for the calculation of partial correlation and standardized regression coefficients , 1985 .

[3]  J. C. Helton,et al.  A COMPARISON OF UNCERTAINTY AND SENSITIVITY ANALYSIS TECHNIQUES FOR COMPUTER MODELS , 1985 .

[4]  James E. Campbell,et al.  An Approach to Sensitivity Analysis of Computer Models: Part I—Introduction, Input Variable Selection and Preliminary Variable Assessment , 1981 .

[5]  Jon C. Helton,et al.  An uncertainty/sensitivity study for the station blackout sequence at a Mark I boiling water reactor , 1989 .

[6]  Terry Andres,et al.  Sensitivity analysis of model output: an investigation of new techniques , 1993 .

[7]  Jay D. Johnson,et al.  Uncertainty and sensitivity analysis of a dry containment test problem for the MAEROS aerosol model , 1989 .

[8]  Ronald L. Iman,et al.  Risk methodology for geologic disposal of radioactive waste: small sample sensitivity analysis techniques for computer models, with an application to risk assessment , 1980 .

[9]  Jon C. Helton,et al.  An Approach to Sensitivity Analysis of Computer Models: Part II - Ranking of Input Variables, Response Surface Validation, Distribution Effect and Technique Synopsis , 1981 .

[10]  Gerald V. Poje.  Resources Needed for New Risk Analysis Opportunities1 , 1988 .

[11]  A. Saltelli,et al.  Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques , 1990 .

[12]  B. M. Brown,et al.  Practical Non-Parametric Statistics. , 1981 .

[13]  R. Iman,et al.  The Use of the Rank Transform in Regression , 1979 .

[14]  J. C. Helton,et al.  An Investigation of Uncertainty and Sensitivity Analysis Techniques for Computer Models , 1988 .