CIRA Guide to Custom Loss Functions for Neural Networks in Environmental Sciences - Version 1

Neural networks are increasingly used in environmental science applications. Furthermore, neural network models are trained by minimizing a loss function, and it is crucial to choose the loss function very carefully for environmental science applications, as it determines what exactly is being optimized. Standard loss functions do not cover all the needs of the environmental sciences, which makes it important for scientists to be able to develop their own custom loss functions so that they can implement many of the classic performance measures already developed in environmental science, including measures developed for spatial model verification. However, there are very few resources available that cover the basics of custom loss function development comprehensively, and to the best of our knowledge none that focus on the needs of environmental scientists. This document seeks to fill this gap by providing a guide on how to write custom loss functions targeted toward environmental science applications. Topics include the basics of writing custom loss functions, common pitfalls, functions to use in loss functions, examples such as fractions skill score as loss function, how to incorporate physical constraints, discrete and soft discretization, and concepts such as focal, robust, and adaptive loss. While examples are currently provided in this guide for Python with Keras and the TensorFlow backend, the basic concepts also apply to other environments, such as Python with PyTorch. Similarly, while the sample loss functions provided here are from meteorology, these are just examples of how to create custom loss functions. Other fields in the environmental sciences have very similar needs for custom loss functions, e.g., for evaluating spatial forecasts effectively, and the concepts discussed here can be applied there as well. All code samples are provided in a GitHub repository. ∗CIRA: Cooperative Institute for Research in the Atmosphere, Colorado State University, Fort Collins, CO. ECE: Electrical and Computer Engineering, Colorado State University, Fort Collins, CO. NOAA-GSL: National Oceanic and Atmospheric Administration (NOAA), Global Systems Laboratory (GSL), Boulder, Colorado. CS: Computer Science, Colorado State University, Fort Collins, CO. CIRES: Cooperative Institute for Research in Environmental Sciences, University of Colorado Boulder, Boulder, CO. ar X iv :2 10 6. 09 75 7v 1 [ cs .L G ] 1 7 Ju n 20 21 CIRA GUIDE TO CUSTOM LOSS FUNCTIONS 1 Why would environmental scientists care about custom loss functions? The use of neural networks in environmental science applications is growing at a rapid pace. In order to train a neural network one has to choose a cost function, called a loss function in the context of neural networks, which represents the error of the neural network. The neural network is then trained, i.e. its parameters are chosen through an iterative process, such that the loss function, and thus the error, is minimized. It is crucial to choose the loss function very carefully for environmental applications, as it determines what exactly the neural network is optimizing. Many pre-defined loss functions exist. The most popular examples for regression (predicting a continuous value such as wind speed) include mean absolute error (MAE), mean squared error (MSE), and root mean squared error (RMSE). The most popular example for classification is cross-entropy. There are many other predefined loss functions, and their number keeps growing, but they do not cover everything environmental scientists care about, since they were developed for other applications. In fact, environmental scientists have a long tradition of developing meaningful performance measures for forecasting tasks, such as for single-category forecasts (accuracy, frequency bias, probability of detection, success ratio, etc.); for multi-category forecasts (Heidke score, Gerrity score, etc.); for continuous forecasts (correlation, reliability diagram, etc.); for probabilistic forecasts (reliability diagram, Brier score, etc.); for spatial forecasts (neighborhood methods, such as fractions skill score, and scale decomposition, such as wavelet decomposition), and many others. For an extremely comprehensive overview of these and other performance measures, see the guide of the WWRP/WGNE Joint Working Group on Forecast Verification Research at https://www.cawcr.gov.au/projects/verification/. However, it is not obvious which ones of those performance measures can easily be used in a neural network and how to do it, for the following reasons: • There are various limitations of what can be implemented in a neural network loss function. Functions must be differentiable and execute extremely quickly, which makes it tricky to implement custom loss functions. • The loss functions required by environmental scientists are unlike any loss functions typically used in computer science, and the community has not yet developed comprehensive resources, such as a large collection of customized loss functions. The above reasons make the topic of loss functions a significant hurdle for practitioners striving to implement meaningful loss functions for their applications. We seek to close this gap here by providing comprehensive instructions, including many examples and discussion of common pitfalls, on how to code custom loss functions. While the sample loss functions provided here are from meteorology, these are just examples of how to create custom loss functions. Other fields in the environmental sciences have very similar needs for custom loss functions, e.g., for evaluating spatial forecasts effectively, and the concepts discussed here can be applied there as well. Making it possible, and even easy, to use a variety of meaningful loss functions will go a long way to more effectively tune neural networks to focus on the types of performance criteria that are truly important in environmental science applications, thus helping the science community to make the most of neural networks for their applications. Scientists in other areas have invested in similar efforts, e.g., researchers in the medical imaging community have developed a collection of loss functions specifically for medical image segmentation [1]. Loss function development for different purposes also remains a very active topic in computer science. See [2, 3, 4, 5] for just a small sample of recent research. 1.1 A case in point: using measures from spatial model verification for neural networks Here we briefly discuss one area with particularly high potential for the development and use of custom loss functions, namely neural networks for spatial forecasts in the environmental sciences. Meteorologists and other environmental scientists have developed an extensive set of evaluation measures for spatial model verification, and ideally those evaluation measures should be used directly for neural network training for forecast models. Why train a neural network on anything other than the criteria we seek to optimize? However, many networks are still trained using pixel-based measures, such as MAE, MSE, or RMSE, mainly because those are easily available as loss functions. Gilleland et al. [6] conducted a large scale comparison of different model verification methods that focus on methods to compare a spatial forecast (image) to an observation (also an image). In [6, 7] they distinguish four primary classes of methods for model verification2: • Neighborhood: Methods that apply some kind of neighborhood averaging to both the forecast and the observation before applying a pixel-based comparison of the resulting smoothed images. Example: Fractions skill score. A later classification by the same group cites five [8], but we prefer the separation into the original four categories described in [6, 7].