Use of neural networks for stable, accurate and physically consistent parameterization of subgrid atmospheric processes with good performance at reduced precision

A promising approach to improve climate-model simulations is to replace traditional subgrid parameterizations based on simplified physical models by machine learning algorithms that are data-driven. However, neural networks (NNs) often lead to instabilities and climate drift when coupled to an atmospheric model. Here we learn an NN parameterization from a high-resolution atmospheric simulation in an idealized domain by coarse graining the model equations and output. The NN parameterization has a structure that ensures physical constraints are respected, and it leads to stable simulations that replicate the climate of the high-resolution simulation with similar accuracy to a successful random-forest parameterization while needing far less memory. We find that the simulations are stable for a variety of NN architectures and horizontal resolutions, and that an NN with substantially reduced numerical precision could decrease computational costs without affecting the quality of simulations.

[1]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[2]  Thomas Bolton,et al.  Applications of Deep Learning to Ocean Data Inference and Subgrid Parameterization , 2019, Journal of Advances in Modeling Earth Systems.

[3]  Noah D. Brenowitz,et al.  Prognostic Validation of a Neural Network Unified Physics Parameterization , 2018, Geophysical Research Letters.

[4]  S. Garner,et al.  Sensitivity of Radiative–Convective Equilibrium Simulations to Horizontal Resolution , 2006 .

[5]  Pierre Gentine,et al.  Deep learning to represent subgrid processes in climate models , 2018, Proceedings of the National Academy of Sciences.

[6]  Noah D. Brenowitz,et al.  Spatially Extended Tests of a Neural Network Parametrization Trained by Coarse‐Graining , 2019, Journal of Advances in Modeling Earth Systems.

[7]  T. Palmer,et al.  More reliable forecasts with less precise computations: a fast-track route to cloud-resolved weather and climate simulators? , 2014, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[8]  Vincent Vanhoucke,et al.  Improving the speed of neural networks on CPUs , 2011 .

[9]  G. Bellon,et al.  The double ITCZ bias in CMIP5 models: interaction between SST, large-scale circulation and precipitation , 2015, Climate Dynamics.

[10]  T. Palmer,et al.  Reduced‐precision parametrization: lessons from an intermediate‐complexity atmospheric model , 2020, Quarterly Journal of the Royal Meteorological Society.

[11]  A. P. Siebesma,et al.  Climate goals and computing the future of clouds , 2017 .

[12]  Noah D. Brenowitz,et al.  Interpreting and Stabilizing Machine-Learning Parametrizations of Convection , 2020, Journal of the Atmospheric Sciences.

[13]  Leslie N. Smith,et al.  Cyclical Learning Rates for Training Neural Networks , 2015, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[14]  Matthew Chantry,et al.  Accelerating High-Resolution Weather Models with Deep-Learning Hardware , 2019, PASC.

[15]  Peter D. Düben,et al.  The use of imprecise processing to improve accuracy in weather & climate prediction , 2014, J. Comput. Phys..

[16]  Prabhat,et al.  Exascale Deep Learning for Climate Analytics , 2018, SC18: International Conference for High Performance Computing, Networking, Storage and Analysis.

[17]  W. Boos,et al.  Response of extreme precipitation to climate warming in quasi-global aquaplanet simulations at high resolution , 2020 .

[18]  Peter D. Düben,et al.  Benchmark Tests for Numerical Weather Forecasts on Inexact Hardware , 2014 .

[19]  Daisuke Miyashita,et al.  Convolutional Neural Networks using Logarithmic Data Representation , 2016, ArXiv.

[20]  W. Boos,et al.  Convective Self-Aggregation and Tropical Cyclogenesis under the Hypohydrostatic Rescaling , 2016 .

[21]  Pierre Gentine,et al.  Could Machine Learning Break the Convection Parameterization Deadlock? , 2018, Geophysical Research Letters.

[22]  Peter D. Düben,et al.  A study of reduced numerical precision to make superparameterization more competitive using a hardware emulator in the OpenIFS model , 2017 .

[23]  Xiaomeng Huang,et al.  A Moist Physics Parameterization Based on Deep Learning , 2020, Journal of Advances in Modeling Earth Systems.

[24]  Vladimir M. Krasnopolsky,et al.  Using Ensemble of Neural Networks to Learn Stochastic Convection Parameterizations for Climate and Numerical Weather Prediction Models from Data Simulated by a Cloud Resolving Model , 2013, Adv. Artif. Neural Syst..

[25]  Pradeep Dubey,et al.  Mixed Precision Training of Convolutional Neural Networks using Integer Operations , 2018, ICLR.

[26]  Pierre Baldi,et al.  A Fortran-Keras Deep Learning Bridge for Scientific Computing , 2020, Sci. Program..

[27]  P. O’Gorman Sensitivity of tropical precipitation extremes to climate change , 2012 .

[28]  W. Boos,et al.  Tropical cyclogenesis in warm climates simulated by a cloud-system resolving model , 2019, Climate Dynamics.

[29]  Pierre Gentine,et al.  Achieving Conservation of Energy in Neural Network Emulators for Climate Modeling , 2019, ArXiv.

[30]  L. Zanna,et al.  Data‐Driven Equation Discovery of Ocean Mesoscale Closures , 2020, Geophysical Research Letters.

[31]  B. Möbis,et al.  Factors controlling the position of the Intertropical Convergence Zone on an aquaplanet , 2012 .

[32]  S. Barnett,et al.  Philosophical Transactions of the Royal Society A : Mathematical , 2017 .

[33]  D. Randall,et al.  Cloud resolving modeling of the ARM summer 1997 IOP: Model formulation, results, uncertainties, and sensitivities , 2003 .

[34]  I. Held,et al.  Resolving Convection in a Global Hypohydrostatic Model , 2007 .

[35]  Christopher S. Bretherton,et al.  A new approach for 3D cloud‐resolving simulations of large‐scale atmospheric circulation , 2005 .

[36]  S. Bony,et al.  Marine boundary layer clouds at the heart of tropical cloud feedback uncertainties in climate models , 2005 .

[37]  Christopher S. Bretherton,et al.  A Mass-Flux Scheme View of a High-Resolution Simulation of a Transition from Shallow to Deep Cumulus Convection , 2006 .

[38]  Yoshua Bengio,et al.  Training deep neural networks with low precision multiplications , 2014 .

[39]  Richard Neale,et al.  A standard test for AGCMs including their physical parametrizations: I: the proposal , 2000 .

[40]  David Patterson,et al.  A domain-specific supercomputer for training deep neural networks , 2020, Commun. ACM.

[41]  P. O'Gorman,et al.  Using Machine Learning to Parameterize Moist Convection: Potential for Modeling of Climate, Climate Change, and Extreme Events , 2018, Journal of Advances in Modeling Earth Systems.

[42]  Hao Wu,et al.  Mixed Precision Training , 2017, ICLR.

[43]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[44]  L. Donner,et al.  The Frequency of Extreme Rain Events in Satellite Rain-Rate Estimates and an Atmospheric General Circulation Model , 2007 .

[45]  Stephan Rasp Coupled online learning as a way to tackle instabilities and biases in neural network parameterizations: general algorithms and Lorenz 96 case study (v1.0) , 2020 .

[46]  Janni Yuval,et al.  Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions , 2020, Nature Communications.

[47]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[48]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.