Problems in RMSE-based wave model validations

Abstract In order to evaluate the reliability of numerical simulations in geophysical applications it is necessary to pay attention when using the root mean square error (RMSE) and two other indicators derived from it (the normalized root mean square error (NRMSE), and the scatter index (SI)). In the present work, in fact, we show on a general basis that, in conditions of constant correlation coefficient, the RMSE index and its variants tend to be systematically smaller (hence identifying better performances of numerical models) for simulations affected by negative bias. Through a geometrical decomposition of RMSE in its components related to the average error and the scatter error it can be shown that the above mentioned behavior is triggered by a quasi-linear dependency between these components in the neighborhood of null bias. This result suggests that smaller values of RMSE, NRMSE and SI do not always identify the best performances of numerical simulations, and that these indicators are not always reliable to assess the accuracy of numerical models. In the present contribution we employ the corrected indicator proposed by Hanna and Heinold (1985) to develop a reliability analysis of wave generation and propagation in the Mediterranean Sea by means of the numerical model WAVEWATCH III®, showing that the best values of the indicator are obtained for simulations unaffected by bias. Evidences suggest that this indicator provides a more reliable information about the accuracy of the results of numerical models.