Evaluating the performance of morphological models

Evaluating the performance of numerical models of coastal morphology against observations is an essential part of establishing their credibility. In the past, this has usually been done by comparing predicted with observed behavior, and the modeler making a subjective judgement of the goodness of fit. By contrast, meteorological models have been tested for many years against observations using objective, scientifically rigorous methods. Drawing on the analogy between distributions of atmospheric pressure (and similar scalars) and coastal bathymetry, the meteorological statistical methods are here adapted to coastal applications. A set of criteria is proposed that identify the desirable attributes of statistical measures of the performance of coastal morphological models. Various statistical parameters used in the meteorological field are described, dealing both with cases in which the model outputs comprise a small number of categories (e.g., advance/equilibrium/retreat of shoreline), and those in which they are continuous (e.g., bathymetry). Examples of the application of these methods are given, in terms of both hypothetical illustrations, and real field data and model predictions. Following meteorological practice, it is shown that measuring the skill of a model (i.e., its performance relative to a simple baseline predictor) is a more critical test than measuring its absolute accuracy. The attributes of the different performance measures are compared with the proposed desirable criteria, and those that match them best are selected. These are the LEPSOB test for categorical data, and the Brier Skill Score (BSS) for continuous data. Routine use of these measures of performance would aid inter-comparability of models, and be a step toward strengthening user confidence in the predictions of coastal numerical models.