Exascale Deep Learning for Climate Analytics

We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parallel efficiency of 79.0%. DeepLabv3+ scales up to 27360 V100 GPUs with a sustained throughput of 325.8 PF/s and a parallel efficiency of 90.7% in single precision. By taking advantage of the FP16 Tensor Cores, a half-precision version of the DeepLabv3+ network achieves a peak and sustained throughput of 1.13 EF/s and 999.0 PF/s respectively.

[1]  Vipin Kumar,et al.  Big Data in Climate: Opportunities and Challenges for Machine Learning , 2016, SIGIR.

[2]  Daniel Walton,et al.  Atmospheric River Tracking Method Intercomparison Project (ARTMIP): project goals and experimental design , 2018, Geoscientific Model Development.

[3]  Takuya Akiba,et al.  Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes , 2017, ArXiv.

[4]  Prabhat,et al.  A deep generative model for astronomical images of galaxies , 2015 .

[5]  Yuanzhou Yang,et al.  Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes , 2018, ArXiv.

[6]  Yann LeCun,et al.  Deep learning with Elastic Averaging SGD , 2014, NIPS.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Kazuhiro Terao,et al.  Machine learning at the energy and intensity frontiers of particle physics , 2018, Nature.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Prabhat,et al.  Application of Deep Convolutional Neural Networks for Detecting Extreme Weather in Climate Datasets , 2016, ArXiv.

[11]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[12]  Sa-Kwang Song,et al.  GlobeNet: Convolutional Neural Networks for Typhoon Eye Tracking from Remote Sensing Imagery , 2017, ArXiv.

[13]  Surendra Byna,et al.  TECA: Petascale Pattern Recognition for Climate Science , 2015, CAIP.

[14]  Oliver Hennigh,et al.  Lat-Net: Compressing Lattice Boltzmann Flow Simulations using Deep Neural Networks , 2017, 1705.09036.

[15]  Yang You,et al.  Large Batch Training of Convolutional Networks , 2017, 1708.03888.

[16]  Prabhat,et al.  CosmoFlow: Using Deep Learning to Learn the Universe at Scale , 2018, SC18: International Conference for High Performance Computing, Networking, Storage and Analysis.

[17]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[18]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[19]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[20]  Yoshua Bengio,et al.  The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Alán Aspuru-Guzik,et al.  Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules , 2016, ACS central science.

[22]  Alexander Sergeev,et al.  Horovod: fast and easy distributed deep learning in TensorFlow , 2018, ArXiv.

[23]  Moritz Müller,et al.  Deep Learning in Science , 2020, ArXiv.

[24]  Takuya Akiba,et al.  ChainerMN: Scalable Distributed Deep Learning Framework , 2017, ArXiv.

[25]  Julia Ling,et al.  Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework , 2017, Physical Review Fluids.

[26]  Surendra Byna,et al.  TECA: A Parallel Toolkit for Extreme Climate Analysis , 2012, ICCS.

[27]  Daniel George,et al.  Deep Neural Networks to Enable Real-time Multimessenger Astrophysics , 2016, ArXiv.