Distinguishing between model- and data-driven inferences for high reliability statistical predictions

Estimating the tails of probability distributions plays a key role in complex engineering systems where the goal is characterization of low probability, high consequence events. When data are collected using physical experimentation, statistical distributional assumptions are often used to extrapolate tail behavior to assess reliability, introducing risk due to extrapolation from an unvalidated (statistical) model. Existing tools to evaluate statistical model fit, such as probability plots and goodness of fit tests, fail to communicate the risk associated with this extrapolation. In this work, we develop a new statistical model validation metric and relate this metric to engineering-driven model validation metrics. The metric measures how consistent the parametric tail estimates are with a more flexible model that makes weaker assumptions about the distribution tails. An extreme-value based generalized Pareto distribution is used for the more flexible model. Models are updated using a Bayesian inference procedure that defaults to reasonably conservative inferences when data are sparse. Properties of the estimation procedure are evaluated in statistical simulation, and the effectiveness of the proposed metrics relative to the standard-of-practice statistical metrics is illustrated using a pedagogical example related to a real, but proprietary, engineering example.