Recurrent Neural Network Architecture Search for Geophysical Emulation

Developing surrogate geophysical models from data is a key research topic in atmospheric and oceanic modeling because of the large computational costs associated with numerical simulation methods. Researchers have started applying a wide range of machine learning models, in particular neural networks, to geophysical data for forecasting without these constraints. Constructing neural networks for forecasting such data is nontrivial, however, and often requires trial and error. To address these limitations, we focus on developing proper-orthogonal-decomposition-based long short-term memory networks (POD-LSTMs). We develop a scalable neural architecture search for generating stacked LSTMs to forecast temperature in the NOAA Optimum Interpolation Sea-Surface Temperature data set. Our approach identifies POD-LSTMs that are superior to manually designed variants and baseline time-series prediction methods. We also assess the scalability of different architecture search strategies on up to 512 Intel Knights Landing nodes of the Theta supercomputer at the Argonne Leadership Computing Facility.

[1]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[2]  Steven L. Brunton,et al.  Multiresolution Dynamic Mode Decomposition , 2015, SIAM J. Appl. Dyn. Syst..

[3]  Bing Xiang,et al.  WeNet: Weighted Networks for Recurrent Network Architecture Search , 2019, ArXiv.

[4]  Michael Maire,et al.  Learning Implicitly Recurrent CNNs Through Parameter Sharing , 2019, ICLR.

[5]  Pierre Gentine,et al.  Deep learning to represent subgrid processes in climate models , 2018, Proceedings of the National Academy of Sciences.

[6]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[7]  Qian Wang,et al.  Benchmark Machine Learning Approaches with Classical Time Series Approaches on the Blood Glucose Level Prediction Challenge , 2018, KHD@IJCAI.

[8]  Steven L. Brunton,et al.  Robust flow reconstruction from limited measurements via sparse representation , 2018, Physical Review Fluids.

[9]  S. Pawar,et al.  A deep learning enabler for nonintrusive reduced order modeling of fluid flows , 2019, Physics of Fluids.

[10]  Gilead Tadmor,et al.  Reduced-Order Modelling for Flow Control , 2013 .

[11]  Fangfang Xia,et al.  Scalable reinforcement-learning-based neural architecture search for cancer deep learning research , 2019, SC.

[12]  A. Mohan,et al.  Compressed Convolutional LSTM: An Efficient Deep Learning framework to Model High Fidelity 3D Turbulence , 2019, 1903.00033.

[13]  D. D. Kosambi Statistics in Function Space , 2016 .

[14]  Traian Iliescu,et al.  A long short-term memory embedding for hybrid uplifted reduced order models , 2019, Physica A: Statistical Mechanics and its Applications.

[15]  Scott T. M. Dawson,et al.  Model Reduction for Flow Analysis and Control , 2017 .

[16]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[17]  Karthik Duraisamy,et al.  Modal Analysis of Fluid Flows: Applications and Outlook , 2019, AIAA Journal.

[18]  P. O'Gorman,et al.  Using Machine Learning to Parameterize Moist Convection: Potential for Modeling of Climate, Climate Change, and Extreme Events , 2018, Journal of Advances in Modeling Earth Systems.

[19]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[20]  Prasanna Balaprakash,et al.  Time-series learning of latent-space dynamics for reduced-order model closure , 2019, Physica D: Nonlinear Phenomena.

[21]  Yuval Tassa,et al.  Emergence of Locomotion Behaviours in Rich Environments , 2017, ArXiv.

[22]  E. Rogers,et al.  The NCEP North American Mesoscale Modeling System : Recent changes and future plans , 2009 .

[23]  Benjamin Peherstorfer,et al.  Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems , 2019, Physica D: Nonlinear Phenomena.

[24]  Fuzhong Weng,et al.  Diagnosis and testing of low-level cloud parameterizations for the NCEP/GFS model using satellite and ground-based measurements , 2013, Climate Dynamics.

[25]  Andrew Stuart,et al.  Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High‐Resolution Simulations , 2017, 1709.00037.

[26]  Devika Subramanian,et al.  Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM , 2019 .

[27]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[28]  P. Holmes,et al.  The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows , 1993 .

[29]  Prasanna Balaprakash,et al.  Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders , 2020, Physics of Fluids.

[30]  Devika Subramanian,et al.  Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM , 2019, ArXiv.

[31]  Ashesh Chattopadhyay,et al.  Analog forecasting of extreme-causing weather patterns using deep learning , 2019, ArXiv.

[32]  Travis Desell,et al.  Investigating recurrent neural network memory structures using neuro-evolution , 2019, GECCO.

[33]  Pierre Gentine,et al.  Could Machine Learning Break the Convection Parameterization Deadlock? , 2018, Geophysical Research Letters.

[34]  Risto Miikkulainen,et al.  From Nodes to Networks: Evolving Recurrent Neural Networks , 2018, ArXiv.

[35]  A. Mohan,et al.  A Deep Learning based Approach to Reduced Order Modeling for Turbulent Flow Control using LSTM Neural Networks , 2018, 1804.09269.

[36]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  K.,et al.  The Community Earth System Model (CESM) large ensemble project: a community resource for studying climate change in the presence of internal climate variability , 2015 .

[38]  Noah D. Brenowitz,et al.  Prognostic Validation of a Neural Network Unified Physics Parameterization , 2018, Geophysical Research Letters.

[39]  G. Powers,et al.  A Description of the Advanced Research WRF Version 3 , 2008 .

[40]  Ashesh Chattopadhyay,et al.  Predicting clustered weather patterns: A test case for applications of convolutional neural networks to spatio-temporal climate data , 2020, Scientific Reports.

[41]  Michael Dellnitz,et al.  Multiobjective Optimal Control Methods for the Navier-Stokes Equations Using Reduced Order Modeling , 2018, Acta Applicandae Mathematicae.

[42]  Steven L. Brunton,et al.  Dynamic Mode Decomposition with Control , 2014, SIAM J. Appl. Dyn. Syst..

[43]  Vassilios Theofilis,et al.  Modal Analysis of Fluid Flows: An Overview , 2017, 1702.01453.

[44]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Qian Wang,et al.  Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem , 2019, J. Comput. Phys..

[47]  Alok Aggarwal,et al.  Regularized Evolution for Image Classifier Architecture Search , 2018, AAAI.