Physics-informed machine learning: case studies for weather and climate modelling

Machine learning (ML) provides novel and powerful ways of accurately and efficiently recognizing complex patterns, emulating nonlinear dynamics, and predicting the spatio-temporal evolution of weather and climate processes. Off-the-shelf ML models, however, do not necessarily obey the fundamental governing laws of physical systems, nor do they generalize well to scenarios on which they have not been trained. We survey systematic approaches to incorporating physics and domain knowledge into ML models and distill these approaches into broad categories. Through 10 case studies, we show how these approaches have been used successfully for emulating, downscaling, and forecasting weather and climate processes. The accomplishments of these studies include greater physical consistency, reduced training time, improved data efficiency, and better generalization. Finally, we synthesize the lessons learned and identify scientific, diagnostic, computational, and resource challenges for developing truly robust and reliable physics-informed ML models for weather and climate processes. This article is part of the theme issue ‘Machine learning for weather and climate modelling’.

[1]  Frans Coenen,et al.  FCNN: Fourier Convolutional Neural Networks , 2017, ECML/PKDD.

[2]  Erik J. Linstead,et al.  A Fortran-Keras Deep Learning Bridge for Scientific Computing , 2020, Sci. Program..

[3]  Michael W. Mahoney,et al.  Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction , 2019, ArXiv.

[4]  E. Tadmor A review of numerical methods for nonlinear partial differential equations , 2012 .

[5]  M. Bonavita,et al.  Machine Learning for Model Error Inference and Correction , 2020, Journal of Advances in Modeling Earth Systems.

[6]  Vipin Kumar,et al.  Integrating Physics-Based Modeling with Machine Learning: A Survey , 2020, ArXiv.

[7]  Jonathan A. Weyn,et al.  Can Machines Learn to Predict Weather? Using Deep Learning to Predict Gridded 500‐hPa Geopotential Height From Historical Weather Data , 2019, Journal of Advances in Modeling Earth Systems.

[8]  Massimo Bonavita,et al.  Using machine learning to correct model error in data assimilation and forecast applications , 2020, ArXiv.

[9]  Wojciech Samek,et al.  Explainable AI: Interpreting, Explaining and Visualizing Deep Learning , 2019, Explainable AI.

[10]  Amy McGovern,et al.  Making the Black Box More Transparent: Understanding the Physical Implications of Machine Learning , 2019, Bulletin of the American Meteorological Society.

[11]  Karthik Kashinath,et al.  MESHFREEFLOWNET: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework , 2020, SC20: International Conference for High Performance Computing, Networking, Storage and Analysis.

[12]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[13]  Amy H. Butler,et al.  A Census of Atmospheric Variability From Seconds to Decades , 2017 .

[14]  Peter Bauer,et al.  Challenges and design choices for global weather and climate models based on machine learning , 2018, Geoscientific Model Development.

[15]  Michael Chertkov,et al.  From Deep to Physics-Informed Learning of Turbulence: Diagnostics , 2018, ArXiv.

[16]  Weiming Hu,et al.  Probabilistic forecasting using deep generative models , 2019, GeoInformatica.

[17]  Patrick Gallinari,et al.  Deep learning for physical processes: incorporating prior scientific knowledge , 2017, ICLR.

[18]  J. Templeton,et al.  Reynolds averaged turbulence modelling using deep neural networks with embedded invariance , 2016, Journal of Fluid Mechanics.

[19]  N. Thürey,et al.  tempoGAN , 2018, ACM Transactions on Graphics.

[20]  Nils Thürey,et al.  tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , 2018, ACM Trans. Graph..

[21]  D. Gagne,et al.  Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model , 2019, Journal of Advances in Modeling Earth Systems.

[22]  Nikola B. Kovachki,et al.  Fourier Neural Operator for Parametric Partial Differential Equations , 2020, ICLR.

[23]  Andrew D. Jones,et al.  Assessing Mountains as Natural Reservoirs With a Multimetric Framework , 2018, Earth's Future.

[24]  Coralia Cartis,et al.  Calibrating climate models using inverse methods: case studies with HadAM3, HadAM3P and HadCM3 , 2017 .

[25]  Malcolm Roberts,et al.  Developing the next-generation climate system models: challenges and achievements , 2009, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[26]  M. Botur,et al.  Lagrangian coherent structures , 2009 .

[27]  Peter A. G. Watson,et al.  Applying Machine Learning to Improve Simulations of a Chaotic Dynamical System Using Empirical Error Correction , 2019, Journal of advances in modeling earth systems.

[28]  P. O'Gorman,et al.  Using Machine Learning to Parameterize Moist Convection: Potential for Modeling of Climate, Climate Change, and Extreme Events , 2018, Journal of Advances in Modeling Earth Systems.

[29]  Steven L. Brunton,et al.  Data-driven discovery of coordinates and governing equations , 2019, Proceedings of the National Academy of Sciences.

[30]  Claire Monteleoni,et al.  ClimAlign: Unsupervised statistical downscaling of climate variables via normalizing flows , 2020, CI.

[31]  The AI revolution in scientific research , 2019 .

[32]  Thomas A. Geddes,et al.  Ensemble deep learning in bioinformatics , 2020, Nature Machine Intelligence.

[33]  Rui Wang,et al.  Incorporating Symmetry into Deep Dynamics Models for Improved Generalization , 2020, ICLR.

[34]  G. Fitzgerald,et al.  'I. , 2019, Australian journal of primary health.

[35]  Pierre Gentine,et al.  Deep learning to represent subgrid processes in climate models , 2018, Proceedings of the National Academy of Sciences.

[36]  A. Chattopadhyay,et al.  Data‐Driven Super‐Parameterization Using Deep Learning: Experimentation With Multiscale Lorenz 96 Systems and Transfer Learning , 2020, Journal of Advances in Modeling Earth Systems.

[37]  Matthias Nießner,et al.  Spherical CNNs on Unstructured Grids , 2019, ICLR.

[38]  Noah D. Brenowitz,et al.  Interpreting and Stabilizing Machine-Learning Parametrizations of Convection , 2020, Journal of the Atmospheric Sciences.

[39]  Alexander Binder,et al.  Unmasking Clever Hans predictors and assessing what machines really learn , 2019, Nature Communications.

[40]  Sebastian Scher,et al.  Weather and climate forecasting with neural networks: using general circulation models (GCMs) with different complexity as a study ground , 2019, Geoscientific Model Development.

[41]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[42]  Soukayna Mouatadid,et al.  WeatherBench: A Benchmark Data Set for Data‐Driven Weather Forecasting , 2020, Journal of Advances in Modeling Earth Systems.

[43]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[44]  Pierre Gentine,et al.  Achieving Conservation of Energy in Neural Network Emulators for Climate Modeling , 2019, ArXiv.

[45]  A. Carrassi,et al.  Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization , 2020, Foundations of Data Science.

[46]  Ramón de Elía,et al.  Objective Calibration of Regional Climate Models: Application over Europe and North America , 2014 .

[47]  Imme Ebert-Uphoff,et al.  Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability , 2019, Journal of Advances in Modeling Earth Systems.

[48]  Towards Physically-Consistent, Data-Driven Models of Convection , 2020, IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium.

[49]  Karthik Kashinath,et al.  Testing the Reliability of Interpretable Neural Networks in Geoscience Using the Madden-Julian Oscillation , 2019 .

[50]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[51]  Pierre Gentine,et al.  Could Machine Learning Break the Convection Parameterization Deadlock? , 2018, Geophysical Research Letters.

[52]  Yaron Lipman,et al.  Invariant and Equivariant Graph Networks , 2018, ICLR.

[53]  Qingming Huang,et al.  Spatiotemporal CNN for Video Object Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Steven L. Brunton,et al.  Deep learning for universal linear embeddings of nonlinear dynamics , 2017, Nature Communications.

[55]  Sebastian Nowozin,et al.  Which Training Methods for GANs do actually Converge? , 2018, ICML.

[56]  Li Li,et al.  Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds , 2018, ArXiv.

[57]  Karthik Duraisamy,et al.  Perspectives on machine learning-augmented Reynolds-averaged and large eddy simulation models of turbulence , 2020, Physical Review Fluids.

[58]  Max Welling,et al.  Gauge Equivariant Convolutional Networks and the Icosahedral CNN 1 , 2019 .

[59]  Anuj Karpatne,et al.  Physics-Guided Architecture (PGA) of Neural Networks for Quantifying Uncertainty in Lake Temperature Modeling , 2019, SDM.

[60]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[61]  Vladimir M. Krasnopolsky,et al.  Using Ensemble of Neural Networks to Learn Stochastic Convection Parameterizations for Climate and Numerical Weather Prediction Models from Data Simulated by a Cloud Resolving Model , 2013, Adv. Artif. Neural Syst..

[62]  Anikesh Pal Deep Learning Emulation of Subgrid‐Scale Processes in Turbulent Shear Flows , 2020, Geophysical Research Letters.

[63]  Anthony J. G. Hey,et al.  The Fourth Paradigm 10 Years On , 2019, Informatik Spektrum.

[64]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Liu Yang,et al.  Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations , 2018, SIAM J. Sci. Comput..

[66]  Karthik Kashinath,et al.  Enforcing Physical Constraints in CNNs through Differentiable PDE Layer , 2020, ICLR 2020.

[67]  G.E. Moore,et al.  Cramming More Components Onto Integrated Circuits , 1998, Proceedings of the IEEE.

[68]  M. F. Kasim,et al.  Building high accuracy emulators for scientific simulations with deep neural architecture search , 2020, Mach. Learn. Sci. Technol..

[69]  Rui Wang,et al.  Towards Physics-informed Deep Learning for Turbulent Flow Prediction , 2019, KDD.

[70]  Stephan Hoyer,et al.  Learning data-driven discretizations for partial differential equations , 2018, Proceedings of the National Academy of Sciences.

[71]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[72]  John Shalf,et al.  The future of computing beyond Moore’s Law , 2020, Philosophical Transactions of the Royal Society A.

[73]  Brian Nord,et al.  Deeply uncertain: comparing methods of uncertainty quantification in deep learning algorithms , 2020, Mach. Learn. Sci. Technol..

[74]  Anuj Karpatne,et al.  Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling , 2017, ArXiv.

[75]  A. Mohan Wavelet-Powered Neural Networks for Turbulence , 2019 .

[76]  Jonathan T. Barron,et al.  Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains , 2020, NeurIPS.

[77]  Yang Yang,et al.  Deep Learning Scaling is Predictable, Empirically , 2017, ArXiv.

[78]  Caskey,et al.  GENERAL CIRCULATION EXPERIMENTS WITH THE PRIMITIVE EQUATIONS I . THE BASIC EXPERIMENT , 1962 .

[79]  Heikki Haario,et al.  Parameter variations in prediction skill optimization at ECMWF , 2013 .

[80]  Todd D. Murphey,et al.  Learning Data-Driven Stable Koopman Operators , 2020, ArXiv.

[81]  David Patterson,et al.  A domain-specific supercomputer for training deep neural networks , 2020, Commun. ACM.

[82]  Janni Yuval,et al.  Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions , 2020, Nature Communications.

[83]  B. L. White,et al.  Downscaling Numerical Weather Models with GANs , 2019 .

[84]  K. Kashinath,et al.  Deep spatial transformers for autoregressive data-driven forecasting of geophysical turbulence , 2020, CI.

[85]  Brandon M. Greenwell,et al.  Interpretable Machine Learning , 2019, Hands-On Machine Learning with R.

[86]  Joachim Denzler,et al.  Deep learning and process understanding for data-driven Earth system science , 2019, Nature.

[87]  C. Hill,et al.  Use of Neural Networks for Stable, Accurate and Physically Consistent Parameterization of Subgrid Atmospheric Processes With Good Performance at Reduced Precision , 2020, Geophysical Research Letters.

[88]  Pascal Fua,et al.  Imposing Hard Constraints on Deep Networks: Promises and Limitations , 2017, CVPR 2017.

[89]  Yaron Lipman,et al.  On the Universality of Invariant Networks , 2019, ICML.

[90]  Imme Ebert-Uphoff,et al.  Evaluation, Tuning, and Interpretation of Neural Networks for Working with Images in Meteorological Applications , 2020 .

[91]  Cynthia Rudin,et al.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , 2018, Nature Machine Intelligence.

[92]  Jennifer G. Dy,et al.  Quantifying Uncertainty in Discrete-Continuous and Skewed Data with Bayesian Deep Learning , 2018, KDD.

[93]  Douglas W. Nychka,et al.  Interpretable Deep Learning for Spatial Analysis of Severe Hailstorms , 2019 .

[94]  Chih-Yuan Yang,et al.  Single-Image Super-Resolution: A Benchmark , 2014, ECCV.

[95]  Adam Rupe,et al.  DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems , 2019, 2019 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC).

[96]  Andrew Stuart,et al.  Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High‐Resolution Simulations , 2017, 1709.00037.

[97]  José Manuel Gutiérrez,et al.  Configuration and intercomparison of deep learning neural models for statistical downscaling , 2019, Geoscientific Model Development.

[98]  Karthik Kashinath,et al.  Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations , 2020, ArXiv.

[99]  Karthik Duraisamy,et al.  Turbulence Modeling in the Age of Data , 2018, Annual Review of Fluid Mechanics.

[100]  Thomas Bolton,et al.  Applications of Deep Learning to Ocean Data Inference and Subgrid Parameterization , 2019, Journal of Advances in Modeling Earth Systems.

[101]  M. Quinlan Living up to the promise , 2004 .

[102]  Adrian Albert,et al.  Emulating Numeric Hydroclimate Models with Physics-Informed cGANs , 2019 .

[103]  P. Sagaut,et al.  Reconstruction of turbulent fluctuations using a hybrid RANS/LES approach , 2002 .

[104]  Kamyar Azizzadenesheli,et al.  Neural Operator: Graph Kernel Network for Partial Differential Equations , 2020, ICLR 2020.

[105]  Andrew Glaws,et al.  Adversarial super-resolution of climatological wind and solar data , 2020, Proceedings of the National Academy of Sciences.

[106]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[107]  D. Klocke,et al.  Tuning the climate of a global model , 2012 .

[108]  Alexander Bihlo,et al.  A generative adversarial network approach to (ensemble) weather prediction , 2020, Neural Networks.

[109]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[110]  Nagiza F. Samatova,et al.  Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data , 2016, IEEE Transactions on Knowledge and Data Engineering.

[111]  Alexandre M. Tartakovsky,et al.  Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks , 2018, J. Comput. Phys..

[112]  Moritz Hardt,et al.  Stable Recurrent Models , 2018, ICLR.

[113]  Michelle Girvan,et al.  Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model , 2018, Chaos.

[114]  Pierre Baldi,et al.  Enforcing Analytic Constraints in Neural Networks Emulating Physical Systems. , 2019, Physical review letters.

[115]  P. Gentine,et al.  Deep learning for subgrid-scale turbulence modeling in large-eddy simulations of the atmospheric boundary layer. , 2019, 1910.12125.

[116]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[117]  Karthik Kashinath,et al.  Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems , 2019, J. Comput. Phys..

[118]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[119]  Paris Perdikaris,et al.  Adversarial Uncertainty Quantification in Physics-Informed Neural Networks , 2018, J. Comput. Phys..

[120]  M. Hutson From models of galaxies to atoms, simple AI shortcuts speed up simulations by billions of times , 2020 .

[121]  Noah D. Brenowitz,et al.  Prognostic Validation of a Neural Network Unified Physics Parameterization , 2018, Geophysical Research Letters.

[122]  William A. Sethares,et al.  SpecNet: Spectral Domain Convolutional Neural Network , 2019, ArXiv.

[123]  Claire A. Miller,et al.  Global lake thermal regions shift under climate change , 2020, Nature Communications.

[124]  Michael Chertkov,et al.  Embedding Hard Physical Constraints in Neural Network Coarse-Graining of 3D Turbulence , 2020, 2002.00021.

[125]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[126]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[127]  Stephan Rasp Coupled online learning as a way to tackle instabilities and biases in neural network parameterizations: general algorithms and Lorenz 96 case study (v1.0) , 2020 .