Proper Evaluation of Neural Network and Learning Systems based Prediction Intervals

Neural Network and Learning Systems approaches are increasingly used in probabilistic prediction. Forecast eval- uation then comprises a complex task for which a number of scores have been proposed, aiming to summarize the assessment of their overall quality with a single number. Such scores ought to be proper though, that is, to effectively reward interval forecasts of higher quality. If not, the ranking of score values does not allow concluding on the actual superiority of a given approach over others, since one may always have the possibility to hedge in order to obtain the best score value. Recently, the Coverage Width-based Criterion (CWC) was proposed and used for an evaluation of the state of the art. The CWC score is shown to be improper based on theoretical considerations, while the consequences are explored.