Survey of sampling-based methods for uncertainty and sensitivity analysis

Sampling-based methods for uncertainty and sensitivity analysis are reviewed. The following topics are considered: (1) Definition of probability distributions to characterize epistemic uncertainty in analysis inputs, (2) Generation of samples from uncertain analysis inputs, (3) Propagation of sampled inputs through an analysis, (4) Presentation of uncertainty analysis results, and (5) Determination of sensitivity analysis results. Special attention is given to the determination of sensitivity analysis results, with brief descriptions and illustrations given for the following procedures/techniques: examination of scatterplots, correlation analysis, regression analysis, partial correlation analysis, rank transformations, statistical tests for patterns based on gridding, entropy tests for patterns based on gridding, nonparametric regression analysis, squared rank differences/rank correlation coefficient test, two dimensional Kolmogorov-Smirnov test, tests for patterns based on distance measures, top down coefficient of concordance, and variance decomposition.

[1]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[2]  A. S. Weigend,et al.  Selecting Input Variables Using Mutual Information and Nonparemetric Density Estimation , 1994 .

[3]  U. Epa,et al.  Guiding Principles for Monte Carlo Analysis , 1997 .

[4]  K. Shuler,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. II Applications , 1973 .

[5]  B. Ripley,et al.  On Sampling Spatial Patterns by Distance Methods , 1980 .

[6]  W. B. Murfin,et al.  The NUREG-1150 probabilistic risk assessment for the Surry Nuclear Power Station☆ , 1992 .

[7]  Robert L. Winkler,et al.  Uncertainty in probabilistic risk assessment , 1996 .

[8]  H. Rabitz,et al.  General foundations of high‐dimensional model representations , 1999 .

[9]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[10]  Marvin K. Nakayama,et al.  Techniques for fast simulation of models of highly dependable systems , 2001, IEEE Trans. Reliab..

[11]  C. Allin Cornell,et al.  Use of Technical Expert Panels: Applications to Probabilistic Seismic Hazard Analysis * , 1998 .

[12]  M. Granger Morgan,et al.  Graphical Communication of Uncertain Quantities to Nontechnical People , 1987 .

[13]  M. Mckay,et al.  Critique of and Limitations on the Use of Expert Judgements in Accident Consequence Uncertainty Analysis , 2000 .

[14]  E. M. Scott.  Modelling Radioactivity in the Environment , 2003 .

[15]  Jon C. Helton,et al.  Conceptual structure of the 1996 performance assessment for the Waste Isolation Pilot Plant , 2000, Reliab. Eng. Syst. Saf..

[16]  K. Byth.  On Robust Distance-Based Intensity Estimators , 1982 .

[17]  André I. Khuri,et al.  Response surface methodology: 1966–1988 , 1989 .

[18]  Jon C. Helton,et al.  A distribution-free test for the relationship between model input and output when using Latin hypercube sampling , 2003, Reliab. Eng. Syst. Saf..

[19]  M. G. Marietta,et al.  Uncertainty and sensitivity analysis results obtained in the 1992 performance assessment for the waste isolation pilot plant , 1996 .

[20]  J. Besag,et al.  Statistical Analysis of Spatial Point Patterns by Means of Distance Methods , 1976 .

[21]  M S Chan,et al.  The consequences of uncertainty for the prediction of the effects of schistosomiasis control programmes , 1996, Epidemiology and Infection.

[22]  K. Bellmann.  Daniel, C., F. S. WOOD, J. W. GORMAN: Fitting Equations to Data. Computer Analysis of Multifactor Data for Scientists and Engineers. John Wiley & Sons, New York-London-Sydney-Toronto 1974. XIV, 342 S., 132 Abb., 33 Tab., £6.50 , 1975 .

[23]  Philip Heidelberger,et al.  A Unified Framework for Simulating Markovian Models of Highly Dependable Systems , 1992, IEEE Trans. Computers.

[24]  M. Eslami,et al.  Introduction to System Sensitivity Theory , 1980, IEEE Transactions on Systems, Man, and Cybernetics.

[25]  J. Peacock.  Two-dimensional goodness-of-fit testing in astronomy , 1983 .

[26]  Peter J. Diggle,et al.  On parameter estimation and goodness-of-fit testing for spatial point patterns , 1979 .

[27]  H. Rabitz,et al.  Efficient input-output model representations , 1999 .

[28]  Martin Berz,et al.  Computational differentiation : techniques, applications, and tools , 1996 .

[29]  F. J. Davis,et al.  Illustration of Sampling‐Based Methods for Uncertainty and Sensitivity Analysis , 2002, Risk analysis : an official publication of the Society for Risk Analysis.

[30]  C. T. Haan.  Parametric Uncertainty in Hydrologic Modeling , 1989 .

[31]  Ilya M. Sobol,et al.  Sensitivity Estimates for Nonlinear Mathematical Models , 1993 .

[32]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[33]  Michael D. McKay,et al.  Evaluating Prediction Uncertainty , 1995 .

[34]  I. Sobol,et al.  Sensitivity Measures, ANOVA-like Techniques and the Use of Bootstrap , 1997 .

[35]  L. Goossens,et al.  Expert judgement for a probabilistic accident consequence uncertainty analysis , 2000 .

[36]  John H. Seinfeld,et al.  Global sensitivity analysis—a computational implementation of the Fourier Amplitude Sensitivity Test (FAST) , 1982 .

[37]  C. Fortuin,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory , 1973 .

[38]  E Ackerman,et al.  Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. IV. Parametric ranges and optimization. , 1993, International journal of bio-medical computing.

[39]  S. Kaplan,et al.  On The Quantitative Definition of Risk , 1981 .

[40]  George J. Klir,et al.  Generalized information theory: aims, results, and open problems , 2004, Reliab. Eng. Syst. Saf..

[41]  Joseph Y. Halpern.  Reasoning about uncertainty , 2003 .

[42]  Jon C. Helton,et al.  A comparison of uncertainty and sensitivity analysis results obtained with random and Latin hypercube sampling , 2005, Reliab. Eng. Syst. Saf..

[43]  Laurence Tianruo Yang,et al.  Fuzzy Logic with Engineering Applications , 1999 .

[44]  Srikanta Mishra,et al.  Application of classification trees in the sensitivity analysis of probabilistic model results , 2003, Reliab. Eng. Syst. Saf..

[45]  L H Goossens,et al.  Joint EC/USNRC expert judgement driven radiological protection uncertainty analysis. , 1998, Journal of radiological protection : official journal of the Society for Radiological Protection.

[46]  P. Diggle,et al.  Some Distance-Based Tests of Independence for Sparsely-Sampled Multivariate Spatial Point Patterns , 1983 .

[47]  Cuthbert Daniel,et al.  Fitting Equations to Data: Computer Analysis of Multifactor Data , 1980 .

[48]  E Ackerman,et al.  Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. II. Durations and peaks. , 1993, International journal of bio-medical computing.

[49]  J. Kacprzyk,et al.  Advances in the Dempster-Shafer theory of evidence , 1994 .

[50]  Terry Andres.  Sampling methods and sensitivity analysis for large parameter sets , 1997 .

[51]  B. Ripley,et al.  Semiparametric Regression: Preface , 2003 .

[52]  T. D. Brown,et al.  The NUREG-1150 probabilistic risk assessment for the Grand Gulf Nuclear Station , 1992 .

[53]  Michael D. McKay,et al.  Nonparametric variance-based methods of assessing uncertainty importance , 1997 .

[54]  Elisabeth Paté-Cornell,et al.  Risk and Uncertainty Analysis in Government Safety Decisions , 2002, Risk analysis : an official publication of the Society for Risk Analysis.

[55]  Philip Heidelberger,et al.  Fast simulation of rare events in queueing and reliability models , 1993, TOMC.

[56]  Nikolay Ivanov Kolev,et al.  Uncertainty and sensitivity analysis of a postexperiment simulation of nonexplosive melt-water interaction , 1996 .

[57]  Raymond H. Myers,et al.  Response Surface Methodology--Current Status and Future Directions , 1999 .

[58]  J. Simonoff.  Smoothing Methods in Statistics , 1998 .

[59]  D. Sharp,et al.  QMU and Nuclear Weapons Certification What ’ s under the hood ? , 2022 .

[60]  C. Granger,et al.  USING THE MUTUAL INFORMATION COEFFICIENT TO IDENTIFY LAGS IN NONLINEAR MODELS , 1994 .

[61]  R. Cooke,et al.  Expert judgement elicitation for risk assessments of critical infrastructures , 2004 .

[62]  Terry Andres,et al.  Sensitivity analysis of model output: an investigation of new techniques , 1993 .

[63]  S. Blower,et al.  Uncertainty and sensitivity analysis of the basic reproductive rate. Tuberculosis as an example. , 1997, American journal of epidemiology.

[64]  Luc Jaulin,et al.  Applied Interval Analysis , 2001, Springer London.

[65]  Efstratios Nikolaidis,et al.  Engineering Design Reliability Handbook , 2004 .

[66]  D. Pontier,et al.  A discrete-event computer model of feline herpes virus within cat populations. , 2000, Preventive Veterinary Medicine.

[67]  R L Sielken,et al.  Challenges to default assumptions stimulate comprehensive realism as a new tier in quantitative cancer risk assessment. , 1995, Regulatory toxicology and pharmacology : RTP.

[68]  Michael Andrew Christie,et al.  Error analysis and simulations of complex phenomena , 2005 .

[69]  J S Evans,et al.  Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. , 1994, Regulatory toxicology and pharmacology : RTP.

[70]  D. Hamby.  A review of techniques for parameter sensitivity analysis of environmental models , 1994, Environmental monitoring and assessment.

[71]  G. E. Apostolakis,et al.  Theoretical foundations and practical issues for using expert judgements in uncertainty analysis of high-level radioactive waste disposal , 1991 .

[72]  A. C. Payne,et al.  The NUREG-1150 probabilistic risk assessment for the Peach Bottom Atomic Power Station , 1992 .

[73]  R. A. Groeneveld,et al.  Practical Nonparametric Statistics (2nd ed). , 1981 .

[74]  B. Ripley.  Spatial Point Pattern Analysis in Ecology , 1987 .

[75]  Jon C. Helton,et al.  Uncertainty and sensitivity analysis for two-phase flow in the vicinity of the repository in the 1996 performance assessment for the Waste Isolation Pilot Plant: undisturbed conditions , 2000, Reliab. Eng. Syst. Saf..

[76]  Norman R. Draper,et al.  Applied regression analysis (2. ed.) , 1981, Wiley series in probability and mathematical statistics.

[77]  William H. Press,et al.  Book-Review - Numerical Recipes in Pascal - the Art of Scientific Computing , 1989 .

[78]  Edward R. Tufte,et al.  The Visual Display of Quantitative Information , 1986 .

[79]  W. V. Loscutoff,et al.  General sensitivity theory , 1972 .

[80]  J. G. Skellam,et al.  A New Method for determining the Type of Distribution of Plant Individuals , 1954 .

[81]  James E. Campbell,et al.  An Approach to Sensitivity Analysis of Computer Models: Part I—Introduction, Input Variable Selection and Preliminary Variable Assessment , 1981 .

[82]  Jon C. Helton,et al.  Uncertainty and sensitivity analysis for two-phase flow in the vicinity of the repository in the 1996 performance assessment for the Waste Isolation Pilot Plant: undisturbed conditions , 2000, Reliab. Eng. Syst. Saf..

[83]  Ronald L. Iman,et al.  Expert opinion in risk analysis: the NUREG-1150 methodology , 1989 .

[84]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[85]  Jon C. Helton,et al.  Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal , 1993 .

[86]  Jon C. Helton,et al.  Latin Hypercube Sampling and the Propagation of Uncertainty in Analyses of Complex Systems , 2002 .

[87]  Richard F. Gunst,et al.  Applied Regression Analysis , 1999, Technometrics.

[88]  Elisabeth Pate ´ Cornell.  Risk and Uncertainty Analysis in Government Safety Decisions , 2002 .

[89]  Joseph A. C. Delaney.  Sensitivity analysis , 2018, The African Continental Free Trade Area: Economic and Distributional Effects.

[90]  Stefano Tarantola,et al.  A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output , 1999, Technometrics.

[91]  H Otway,et al.  Expert judgment in risk analysis and management: process, context, and pitfalls. , 1992, Risk analysis : an official publication of the Society for Risk Analysis.

[92]  Wallace B. Whiting,et al.  Effect of uncertainties in thermodynamic data and model parameters on calculated process performance , 1993 .

[93]  Ronald L. Iman,et al.  FORTRAN 77 program and user's guide for the calculation of partial correlation and standardized regression coefficients , 1985 .

[94]  J. Ord,et al.  Spatial and temporal analysis in ecology , 1981 .

[95]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[96]  M. Elisabeth Paté-Cornell,et al.  Uncertainties in risk analysis: Six levels of treatment , 1996 .

[97]  J. C. Helton,et al.  Statistical Analyses of Scatterplots to Identify Important Factors in Large-Scale Simulations , 1999 .

[98]  J. Couper.  Sensitivity and Uncertainty Analysis , 2003 .

[99]  Eduard Hofer,et al.  Sensitivity analysis in the context of uncertainty analysis for computationally intensive models , 1999 .

[100]  R Mead,et al.  A review of response surface methodology from a biometric viewpoint. , 1975, Biometrics.

[101]  W. B. Murfin,et al.  The NUREG-1150 probabilistic risk assessment for the Sequoyah Nuclear Plant☆ , 1992 .

[102]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[103]  M. Jansen,et al.  Monte Carlo estimation of uncertainty contributions from several independent multivariate sources. , 1994 .

[104]  M. Jansen.  Analysis of variance designs for model output , 1999 .

[105]  Jack P. C. Kleijnen,et al.  Sensitivity analysis and related analyses: A review of some statistical techniques , 1997 .

[106]  D. Cacuci,et al.  A Comparative Review of Sensitivity and Uncertainty Analysis of Large-Scale Systems—II: Statistical Methods , 2004 .

[107]  E Ackerman,et al.  Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. I. Illness attack rates. , 1993, International journal of bio-medical computing.

[108]  Eric E. Smith,et al.  Uncertainty analysis , 2001 .

[109]  J. Kleijnen,et al.  Statistical analyses of scatterplots to identify important factors in large-scale simulations, 2: robustness of techniques , 1999 .

[110]  F. O. Hoffman,et al.  Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. , 1994, Risk analysis : an official publication of the Society for Risk Analysis.

[111]  M. Becker,et al.  Sensitivity and uncertainty analysis of reactor performance parameters , 1982 .

[112]  Guangzhou Zeng,et al.  A comparison of tests for randomness , 1985, Pattern Recognit..

[113]  Jon C. Helton,et al.  Treatment of Uncertainty in Performance Assessments for Complex Systems , 1994 .

[114]  T. M. CHARLTON.  Civil Engineering , 1961, Nature.

[115]  Jon C. Helton,et al.  Representation of two-phase flow in the vicinity of the repository in the 1996 performance assessment for the Waste Isolation Pilot Plant , 2000, Reliab. Eng. Syst. Saf..

[116]  George Apostolakis,et al.  A taxonomy of issues related to the use of expert judgments in probabilistic safety studies , 1992 .

[117]  H. Rabitz,et al.  High Dimensional Model Representations , 2001 .

[118]  Connie M. Borror,et al.  Response Surface Methodology: A Retrospective and Literature Survey , 2004 .

[119]  R. C. MacDonald,et al.  Valuation of Supplemental and Enhanced Oil Recovery Projects With Risk Analysis , 1986 .

[120]  Mike Rees,et al.  5. Statistics for Spatial Data , 1993 .

[121]  R. Iman,et al.  Rank correlation plots for use with correlated input variables , 1982 .

[122]  Peter A. W. Lewis,et al.  Stochastic point processes : statistical analysis, theory, and applications , 1973 .

[123]  Alex H. Barbat,et al.  Monte Carlo techniques in computational stochastic mechanics , 1998 .

[124]  Hal Caswell,et al.  HARBOR PORPOISE AND FISHERIES: AN UNCERTAINTY ANALYSIS OF INCIDENTAL MORTALITY , 1998 .

[125]  Xiaobo Zhou,et al.  Global Sensitivity Analysis , 2017, Encyclopedia of GIS.

[126]  M. Morris.  Three Technometrics experimental design classics , 2000 .

[127]  Trevor Hastie,et al.  Statistical Models in S , 1991 .

[128]  Jack P. C. Kleijnen,et al.  Sensitivity analysis of simulation experiments: regression analysis and statistical design , 1992 .

[129]  J. Garvey,et al.  FROM STAR CHARTS TO STONEFLIES: DETECTING RELATIONSHIPS IN CONTINUOUS BIVARIATE DATA , 1998 .

[130]  R. Moddemeijer.  On estimation of entropy and mutual information of continuous distributions , 1989 .

[131]  A. Owen,et al.  Safe and Effective Importance Sampling , 2000 .

[132]  M. Kramer,et al.  Sensitivity Analysis in Chemical Kinetics , 1983 .

[133]  ' RonaldL.Iman,et al.  An Investigation of Uncertainty and Sensitivity Analysis Techniques for Computer Models , 2006 .

[134]  R. Iman,et al.  A measure of top-down correlation , 1987 .

[135]  T. Cox,et al.  A conditioned distance ratio method for analyzing spatial patterns , 1976 .

[136]  Hadi Dowlatabadi,et al.  Sensitivity and Uncertainty Analysis of Complex Models of Disease Transmission: an HIV Model, as an Example , 1994 .

[137]  D. Cacuci,et al.  A Comparative Review of Sensitivity and Uncertainty Analysis of Large-Scale Systems—I: Deterministic Methods , 2004 .

[138]  Laura Toran,et al.  Subsurface stormflow modeling with sensitivity analysis using a Latin-hypercube sampling technique , 1996 .

[139]  A. Saltelli,et al.  Sensitivity Anaysis as an Ingredient of Modeling , 2000 .

[140]  I. Sobol,et al.  About the use of rank transformation in sensitivity analysis of model output , 1995 .

[141]  J. D. Morrison,et al.  Evaluating prediction uncertainty in simulation models , 1999 .

[142]  Peter J. Diggle,et al.  Simple Monte Carlo Tests for Spatial Pattern , 1977 .

[143]  Nathan Siu,et al.  Bayesian parameter estimation in probabilistic risk assessment , 1998 .

[144]  Jon C. Helton,et al.  Guest editorial: treatment of aleatory and epistemic uncertainty in performance assessments for complex systems , 1996 .

[145]  Ronald L. Iman,et al.  Risk methodology for geologic disposal of radioactive waste: small sample sensitivity analysis techniques for computer models, with an application to risk assessment , 1980 .

[146]  George J. Klir,et al.  Uncertainty-Based Information , 1999 .

[147]  Tim B. Swartz,et al.  Approximating Integrals Via Monte Carlo and Deterministic Methods , 2000 .

[148]  Jon C. Helton,et al.  An Approach to Sensitivity Analysis of Computer Models: Part II - Ranking of Input Variables, Response Surface Validation, Distribution Effect and Technique Synopsis , 1981 .

[149]  Jon C. Helton,et al.  Summary description of the methods used in the probabilistic risk assessments for NUREG-1150 , 1992 .

[150]  H. Gershengorn,et al.  A tale of two futures: HIV and antiretroviral therapy in San Francisco. , 2000, Science.

[151]  Perwez Shahabuddin,et al.  Importance sampling for the simulation of highly reliable Markovian systems , 1994 .

[152]  P. J. Clark,et al.  Distance to Nearest Neighbor as a Measure of Spatial Relationships in Populations , 1954 .

[153]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice , 2002 .

[154]  R. Cooke.  Elicitation of expert opinions for uncertainty and risks , 2003 .

[155]  J. Neter,et al.  Applied linear statistical models : regression, analysis of variance, and experimental designs , 1974 .

[156]  Jon C. Helton,et al.  Calculation of reactor accident safety goals , 1993 .

[157]  G. A. Barnard,et al.  Discussion of Professor Bartlett''s paper , 1963 .

[158]  Jon C. Helton,et al.  Multiple predictor smoothing methods for sensitivity analysis , 2005, Proceedings of the Winter Simulation Conference, 2005..

[159]  Patrick J. Roache,et al.  Verification and Validation in Computational Science and Engineering , 1998 .

[160]  G. W. Parry,et al.  Characterization and evaluation of uncertainty in probabilistic risk analysis , 1981 .

[161]  R. Iman,et al.  The Use of the Rank Transform in Regression , 1979 .

[162]  G. Apostolakis.  The concept of probability in safety assessments of technological systems. , 1990, Science.

[163]  Michael C. Cheok,et al.  An approach for using risk assessment in risk-informed decisions on plant-specific changes to the licensing basis , 1999 .

[164]  William H. Press,et al.  Numerical Recipes in FORTRAN - The Art of Scientific Computing, 2nd Edition , 1987 .

[165]  Donald L. Iglehart,et al.  Importance sampling for stochastic simulations , 1989 .

[166]  Ola Svenson,et al.  On expert judgements in safety analyses in the process industries , 1989 .

[167]  M. Thorne,et al.  A review of expert judgment techniques with reference to nuclear safety , 1992 .

[168]  Jon C. Helton,et al.  Direct releases to the surface and associated complementary cumulative distribution functions in the 1996 performance assessment for the Waste Isolation Pilot Plant: cuttings, cavings and spallings , 2000, Reliab. Eng. Syst. Saf..

[169]  Stan Kaplan,et al.  ‘Expert information’ versus ‘expert opinions’. Another approach to the problem of eliciting/ combining/using expert knowledge in PRA , 1992 .

[170]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[171]  F. W. Whicker,et al.  Contaminant Transport through Agroecosystems: Assessing Relative Importance of Environmental, Physiological, and Management Factors. , 1992, Ecological applications : a publication of the Ecological Society of America.

[172]  J. Charles Kerkering,et al.  Eliciting and Analyzing Expert Judgment, A Practical Guide , 2002, Technometrics.

[173]  K.,et al.  Nonlinear sensitivity analysis of multiparameter model systems , 1977 .

[174]  A. Saltelli,et al.  An alternative way to compute Fourier amplitude sensitivity test (FAST) , 1998 .

[175]  Jane M. Booker,et al.  Fuzzy Logic and Probability Applications , 2002, Fuzzy Logic and Probability Applications.

[176]  Jack P. C. Kleijnen,et al.  An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis , 2005, Eur. J. Oper. Res..

[177]  Jon C. Helton,et al.  Summary discussion of the 1996 performance assessment for the Waste Isolation Pilot Plant , 2000, Reliab. Eng. Syst. Saf..

[178]  Olaf Wolkenhauer.  Data engineering - fuzzy mathematics in systems theory and data analysis , 2001 .

[179]  Jon C. Helton,et al.  Chapter 12 Mathematical and numerical approaches in performance assessment for radioactive waste disposal: dealing with uncertainty , 2000 .

[180]  G. Fasano,et al.  A multidimensional version of the Kolmogorov–Smirnov test , 1987 .

[181]  Jon C. Helton,et al.  Uncertainty and Sensitivity Analysis Results Obtained in the 1996 Performance Assessment for the Waste Isolation Pilot Plant , 1998 .

[182]  J. C. Helton,et al.  Statistical Analyses of Scatterplots to Identify Important Factors in Large-Scale Simulations, 1: Review and Comparison of Techniques , 1999 .

[183]  Edward H. Shortliffe,et al.  The Dempster-Shafer theory of evidence , 1990 .

[184]  P. Holgate.  Tests of randomness based on distance methods , 1965 .

[185]  R. Iman,et al.  A distribution-free approach to inducing rank correlation among input variables , 1982 .

[186]  Robert Haining,et al.  Statistics for spatial data: by Noel Cressie, 1991, John Wiley & Sons, New York, 900 p., ISBN 0-471-84336-9, US $89.95 , 1993 .

[187]  Bilal M. Ayyub,et al.  Uncertainty Modeling and Analysis in Civil Engineering , 1997 .

[188]  R. Zeckhauser,et al.  The perils of prudence: how conservative risk assessments distort regulation. , 1988, Regulatory toxicology and pharmacology : RTP.

[189]  H Christopher Frey,et al.  OF SENSITIVITY ANALYSIS , 2001 .

[190]  尚弘 島影.  National Institute of Standards and Technologyにおける超伝導研究及び生活 , 2001 .

[191]  Stefano Tarantola,et al.  Winding Stairs: A sampling tool to compute sensitivity indices , 2000, Stat. Comput..

[192]  Ali Mosleh,et al.  A critique of current practice for the use of expert opinions in probabilistic risk assessment , 1988 .

[193]  A. Saltelli,et al.  Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques , 1990 .

[194]  R. H. Myers.  Classical and modern regression with applications , 1986 .

[195]  Jon C. Helton,et al.  Characterization of subjective uncertainty in the 1996 performance assessment for the Waste Isolation Pilot Plant , 2000, Reliab. Eng. Syst. Saf..

[196]  Jon C. Helton,et al.  An exploration of alternative approaches to the representation of uncertainty in model predictions , 2003, Reliab. Eng. Syst. Saf..

[197]  B. Ripley.  Tests of 'Randomness' for Spatial Point Patterns , 1979 .

[198]  Jon C. Helton,et al.  The 1996 performance assessment for the Waste Isolation Pilot Plant , 1998, Reliability Engineering & System Safety.

[199]  Jon C. Helton,et al.  Sampling-based methods for uncertainty and sensitivity analysis. , 2000 .

[200]  J. C. Helton,et al.  Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty , 1997 .

[201]  Kathleen V. Diegert,et al.  Error and uncertainty in modeling and simulation , 2002, Reliab. Eng. Syst. Saf..

[202]  R. Melchers.  Search-based importance sampling , 1990 .

[203]  A. Saltelli,et al.  Importance measures in global sensitivity analysis of nonlinear models , 1996 .

[204]  T. Turányi.  Sensitivity analysis of complex kinetic systems. Tools and applications , 1990 .

[205]  D. Cacuci,et al.  SENSITIVITY and UNCERTAINTY ANALYSIS , 2003 .

[206]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[207]  Jürgen Symanzik,et al.  Statistical Analysis of Spatial Point Patterns , 2005, Technometrics.