Convergence of artificial intelligence and high performance computing on NSF-supported cyberinfrastructure

Introduction The big data revolution disrupted the digital and computing landscape in the early 2010s [1]. Data torrents produced by corporations such as Google, Amazon, Facebook and YouTube, among others, presented a unique opportunity for innovation. Traditional signal processing tools and computing methodologies were inadequate to turn these big-data challenges into technological breakthroughs. A radical rethinking was urgently needed [2, 3]. Large Scale Visual Recognition Challenges [4] set the scene for the ongoing digital revolution. The quest for novel pattern recognition algorithms [5–7] that sift through large, Abstract

[1]  Yuanzhou Yang,et al.  Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes , 2018, ArXiv.

[2]  E. Huerta,et al.  Artificial neural network subgrid models of 2D compressible magnetohydrodynamic turbulence , 2019, Physical Review D.

[3]  T. Roche,et al.  Industry , 1995 .

[4]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[5]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[6]  Ian T. Foster,et al.  DLHub: Model and Data Serving for Science , 2018, 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[7]  E. A. Huerta,et al.  Physics-inspired deep learning to characterize the signal manifold of quasi-circular, spinning, non-precessing binary black hole mergers , 2020, ArXiv.

[8]  Volodymyr Kindratenko,et al.  Review and Examination of Input Feature Preparation Methods and Machine Learning Models for Turbulence Modeling. , 2020 .

[9]  Kyle Chard,et al.  A data ecosystem to support machine learning in materials science , 2019, MRS Communications.

[10]  Daniel Cremers,et al.  Regularization for Deep Learning: A Taxonomy , 2017, ArXiv.

[11]  D. Whiteson,et al.  Deep Learning and Its Application to LHC Physics , 2018, Annual Review of Nuclear and Particle Science.

[12]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[13]  E. Huerta,et al.  Deep Learning at Scale for the Construction of Galaxy Catalogs in the Dark Energy Survey. , 2019 .

[14]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[15]  Yan Zhao,et al.  Clowder: Open Source Data Management for Long Tail Data , 2018, PEARC.

[16]  Telecommunications Board.  Opportunities from the Integration of Simulation Science and Data Science: Proceedings of a Workshop , 2018 .

[17]  Hongyu Shen,et al.  Enabling real-time multi-messenger astrophysics discoveries with deep learning , 2019, Nature Reviews Physics.

[18]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[19]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Alexander Sergeev,et al.  Horovod: fast and easy distributed deep learning in TensorFlow , 2018, ArXiv.

[21]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[22]  E. al.,et al.  The Sloan Digital Sky Survey: Technical summary , 2000, astro-ph/0006396.

[23]  Gregory M. Kurtzer,et al.  Singularity 2.1.2 - Linux application and environment containers for science , 2016 .

[24]  S. Huber,et al.  Learning phase transitions by confusion , 2016, Nature Physics.

[25]  Achille Fokoue,et al.  An effective algorithm for hyperparameter optimization of neural networks , 2017, IBM J. Res. Dev..

[26]  Hongyu Shen,et al.  Deep Learning at Scale for Gravitational Wave Parameter Estimation of Binary Black Hole Mergers , 2019, ArXiv.

[27]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[28]  Jiawei Han,et al.  Knowledge-guided analysis of "omics" data using the KnowEnG cloud platform , 2019, bioRxiv.

[29]  William Gropp,et al.  HAL: Computer System for Scalable Deep Learning , 2020, PEARC.

[30]  Seid Koric,et al.  Machine learning accelerated topology optimization of nonlinear structures , 2020, ArXiv.

[31]  Rajeev S. Assary,et al.  Machine learning prediction of accurate atomization energies of organic molecules from low-fidelity quantum chemical calculations , 2019, MRS Communications.

[32]  James Demmel,et al.  ImageNet Training in Minutes , 2017, ICPP.

[33]  Michael Carbin,et al.  The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks , 2018, ICLR.

[34]  Rui Liu,et al.  Brown Dog: Leveraging everything towards autocuration , 2015, 2015 IEEE International Conference on Big Data (Big Data).

[35]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[38]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[39]  Terrence J Sejnowski,et al.  The unreasonable effectiveness of deep learning in artificial intelligence , 2020, Proceedings of the National Academy of Sciences.

[40]  Julian Kates-Harbeck,et al.  Training distributed deep recurrent neural networks with mixed precision on GPU clusters , 2017, MLHPC@SC.

[41]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[42]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[43]  Prasanna Balaprakash,et al.  DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks , 2018, 2018 IEEE 25th International Conference on High Performance Computing (HiPC).