From SIAM News , Volume 43 , Number 10 , December 2010 Computer Predictions with Quantified Uncertainty , Part II

The Validation Process To begin this process, we need a new set of observations, designed to challenge the model. The new observational data and their uncertainties will then be compared to the “predictions” of the model and its uncertainties for the physical scenarios in which the observations were made (the “validation scenarios”). Here, however, we encounter a problem: The validation observations are generally not of the quantities of interest (QoI’s) or for the scenarios of the prediction, but rather for some validation quantities that are experimentally accessible in the validation scenarios. So how are the results of the comparison between model and validation experiments to be evaluated? What level of disagreement can be tolerated for the purposes of the predictions to be made? The answers to these questions depend critically on the character of the prediction problem at hand. First, we note that the validation questions posed here are somewhat different from those arising in the usual experimental falsification of scientific theories. In the latter, the aim is to discern the laws of nature (theories); a disagreement between theory and experiment thus falsifies the theory. Or, in the context of uncertainty, a low probability that the experimental observations are consistent with the theory makes the theory improbable. We know, for example, that measurements taken in relativistic scenarios will falsify Newtonian mechanics as a scientific theory. But for our present purposes, we seek useful models, which may well be imperfect, and ask a narrower question: Are the models valid for making particular predictions? To return to our example: Newtonian mechanics, despite having been falsified as a theory, provides a valid model for making many predictions. Unfortunately, models used for making predictions in complex systems generally do not have the wellcharacterized domains of applicability of Newtonian mechanics. Validation processes, as discussed here, are thus needed to detect whether a model proposed for use in a particular prediction is invalid for that purpose. At the heart of the validation process is the question: What do discrepancies between models and experimental observations imply about the reliability of the specific QoI predictions to be made by the models? To address this question, Bayes’s theorem can again be used, this time to update the model and its parameters in light of the validation observations. The effect of the updates on the predictions can then be determined by using the updated model to make predictions of the QoI’s. If the change in the predictions is too large in some appropriate measure (as discussed below), then the inconsistency of the model with the validation data will influence the predictions, and the model and its calibration are invalid for predicting those QoI’s.* A flow chart illustrating an example of this validation approach is shown in Figure 2. In this process, the consequences of the validation observations must be representable within the structure of the model. In some situations, the models have a parametrization sufficiently rich to permit representation of almost any observation. In such cases, the validation data can be used to update the model parameters, and the invalidity of the model would be reflected in the inconsistency of the parameters needed to represent the validation data (relative to the calibration) and the impact of the inconsistency on the predicted QoI’s. When such a parametric representation is not possible, the model will need to be enriched so that it can represent the validation data. One way to do this is to introduce a statistical model of the model error, and use it to represent the discrepancies between model and validation data. The impact of the discrepancy model on the QoI’s, and thus the possible invalidity of the original model, can then be determined. The impact of the validation observations on the predicted QoI’s is expressed as the differences between two probability densities for the QoI’s. Assessing the invalidity of the model then rests on an appropriate metric and tolerance for differences in these distributions. What this metric should be depends on the questions asked about the QoI’s. Perhaps we need to know the most likely value of a QoI with a specified tolerance for error, in which Figure 2. Schematic of an example of the validation process.