86 PFLOPS Deep Potential Molecular Dynamics simulation of 100 million atoms with ab initio accuracy

Abstract We present the GPU version of DeePMD-kit, which, upon training a deep neural network model using ab initio data, can drive extremely large-scale molecular dynamics (MD) simulation with ab initio accuracy. Our tests show that for a water system of 12 , 582 , 912 atoms, the GPU version can be 7 times faster than the CPU version under the same power consumption. The code can scale up to the entire Summit supercomputer. For a copper system of 113 , 246 , 208 atoms, the code can perform one nanosecond MD simulation per day, reaching a peak performance of 86 PFLOPS (43% of the peak). Such unprecedented ability to perform MD simulation with ab initio accuracy opens up the possibility of studying many important issues in materials and molecules, such as heterogeneous catalysis, electrochemical cells, irradiation damage, crack propagation, and biochemical reactions. Program summary Program Title: DeePMD-kit CPC Library link to program files: https://doi.org/10.17632/phyn4kgsfx.1 Developer’s repository link: https://doi.org/10.5281/zenodo.3961106 Licensing provisions: LGPL Programming language: C++/Python/CUDA Journal reference of previous version: Comput. Phys. Commun. 228 (2018), 178–184. Does the new version supersede the previous version?: Yes. Reasons for the new version: Parallelize and optimize the DeePMD-kit for modern high performance computers. Summary of revisions: The optimized DeePMD-kit is capable of computing 100 million atoms molecular dynamics with ab initio accuracy, achieving 86 PFLOPS in double precision. Nature of problem: Modeling the many-body atomic interactions by deep neural network models. Running molecular dynamics simulations with the models. Solution method: The Deep Potential for Molecular Dynamics (DeePMD) method is implemented based on the deep learning framework TensorFlow. Standard and customized TensorFlow operators are optimized for GPU. Massively parallel molecular dynamics simulations with DeePMD models on high performance computers are supported in the new version.

[1]  Wei-Hai Fang,et al.  Deep Learning for Nonadiabatic Excited-State Dynamics. , 2018, The journal of physical chemistry letters.

[2]  S. Goedecker.  Linear scaling electronic structure methods , 1999 .

[3]  David W Toth,et al.  The TensorMol-0.1 model chemistry: a neural network augmented with long-range physics , 2017, Chemical science.

[4]  Michele Parrinello,et al.  Silicon Liquid Structure and Crystal Nucleation from Ab Initio Deep Metadynamics. , 2018, Physical review letters.

[5]  John A. Gunnels,et al.  Beyond homogeneous decomposition: scaling long-range forces on Massively Parallel Systems , 2009, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis.

[6]  Paolo Bientinesi,et al.  The Vectorization of the Tersoff Multi-body Potential: An Exercise in Performance Portability , 2016, SC16: International Conference for High Performance Computing, Networking, Storage and Analysis.

[7]  J. P. Grossman,et al.  Anton 2: Raising the Bar for Performance and Programmability in a Special-Purpose Molecular Dynamics Supercomputer , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[8]  Paolo Bientinesi,et al.  LAMMPS' PPPM Long-Range Solver for the Second Generation Xeon Phi , 2017, ISC.

[9]  E Weinan,et al.  Active Learning of Uniformly Accurate Inter-atomic Potentials for Materials Simulation , 2018, Physical Review Materials.

[10]  R. Car,et al.  Free energy of proton transfer at the water–TiO2 interface from ab initio deep potential molecular dynamics† , 2020, Chemical science.

[11]  Robert S. Germain,et al.  Blue Matter: Strong Scaling of Molecular Dynamics on Blue Gene/L , 2006, International Conference on Computational Science.

[12]  E. Weinan,et al.  Deep Potential: a general representation of a many-body potential energy surface , 2017, 1707.01478.

[13]  T. Koishi,et al.  An 8.61 Tflop/s Molecular Dynamics Simulation for NaCl with a Special-Purpose Computer: MDM , 2000, ACM/IEEE SC 2001 Conference (SC'01).

[14]  John A. Gunnels,et al.  Extending stability beyond CPU millennium: a micron-scale atomistic simulation of Kelvin-Helmholtz instability , 2007, Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC '07).

[15]  Erich Strohmaier,et al.  Linearly scaling 3D fragment method for large-scale electronic structure calculations , 2008, HiPC 2008.

[16]  Markus Eisenbach,et al.  A scalable method for ab initio computation of free energies in nanoscale systems , 2009, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis.

[17]  Pak Lui,et al.  Strong scaling of general-purpose molecular dynamics simulations on GPUs , 2014, Comput. Phys. Commun..

[18]  E Weinan,et al.  Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics , 2017, Physical review letters.

[19]  Klaus-Robert Müller,et al.  Machine learning of accurate energy-conserving molecular force fields , 2016, Science Advances.

[20]  Michele Parrinello,et al.  Generalized neural-network representation of high-dimensional potential-energy surfaces. , 2007, Physical review letters.

[21]  Seungwu Han,et al.  SIMPLE-NN: An efficient package for training and executing neural-network interatomic potentials , 2019, Comput. Phys. Commun..

[22]  Christoph Dellago,et al.  Library-Based LAMMPS Implementation of High-Dimensional Neural Network Potentials. , 2019, Journal of chemical theory and computation.

[23]  Linfeng Zhang,et al.  Deep learning inter-atomic potential model for accurate irradiation damage simulations , 2019, Applied Physics Letters.

[24]  Carsten Kutzner,et al.  GROMACS 4:  Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation. , 2008, Journal of chemical theory and computation.

[25]  J. P. Grossman,et al.  Anton, a special-purpose machine for molecular dynamics simulation , 2008, CACM.

[26]  Aris Marcolongo,et al.  Simulating Diffusion Properties of Solid‐State Electrolytes via a Neural Network Potential: Performance and Training Scheme , 2019, ChemSystemsChem.

[27]  Holger Gohlke,et al.  The Amber biomolecular simulation programs , 2005, J. Comput. Chem..

[28]  Toshikazu Ebisuzaki,et al.  1.34 Tflops Molecular Dynamics Simulation for NaCl with a Special-Purpose Computer: MDM , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[29]  Car,et al.  Unified approach for molecular dynamics and density-functional theory. , 1985, Physical review letters.

[30]  D. van der Spoel,et al.  GROMACS: A message-passing parallel molecular dynamics implementation , 1995 .

[31]  Alireza Khorshidi,et al.  Amp: A modular approach to machine learning in atomistic simulations , 2016, Comput. Phys. Commun..

[32]  E Weinan,et al.  End-to-end Symmetry Preserving Inter-atomic Potential Energy Model for Finite and Extended Systems , 2018, NeurIPS.

[33]  E Weinan,et al.  DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics , 2017, Comput. Phys. Commun..

[34]  Masanori Hariyama,et al.  Architecture of an FPGA accelerator for molecular dynamics simulation using OpenCL , 2016, 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS).

[35]  Berk Hess,et al.  A flexible algorithm for calculating pair interactions on SIMD architectures , 2013, Comput. Phys. Commun..

[36]  Peng Wang,et al.  Implementing molecular dynamics on hybrid high performance computers - short range forces , 2011, Comput. Phys. Commun..

[37]  Yanchun Zhou,et al.  Theoretical prediction on thermal and mechanical properties of high entropy (Zr0.2Hf0.2Ti0.2Nb0.2Ta0.2)C by deep learning potential , 2020 .

[38]  Ryutaro Himeno,et al.  A 55 TFLOPS simulation of amyloid-forming peptides from yeast prion Sup35 with the special-purpose computer system MDGRAPE-3 , 2006, SC.

[39]  Mohan Chen,et al.  Structure and dynamics of warm dense aluminum: a molecular dynamics study with density functional theory and deep potential , 2019, Journal of physics. Condensed matter : an Institute of Physics journal.

[40]  Joshua A. Anderson,et al.  General purpose molecular dynamics simulations fully implemented on graphics processing units , 2008, J. Comput. Phys..

[41]  Daniel G A Smith,et al.  PES-Learn: An Open-Source Software Package for the Automated Generation of Machine Learning Models of Molecular Potential Energy Surfaces. , 2019, Journal of chemical theory and computation.

[42]  Tong Zhu,et al.  Neural Network Based in Silico Simulation of Combustion Reactions , 2019, ArXiv.

[43]  Jiayi Sheng,et al.  Fully Integrated On-FPGA Molecular Dynamics Simulations , 2019, ArXiv.

[44]  Linfeng Zhang,et al.  DeePCG: Constructing coarse-grained models via deep neural networks. , 2018, The Journal of chemical physics.

[45]  Arnold N. Tharrington,et al.  High-Performance Molecular Dynamics Simulation for Biological and Materials Sciences: Challenges of Performance Portability , 2018, 2018 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC).

[46]  Klaus-Robert Müller,et al.  SchNet: A continuous-filter convolutional neural network for modeling quantum interactions , 2017, NIPS.

[47]  J S Smith,et al.  ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost , 2016, Chemical science.

[48]  Duncan Poole,et al.  Routine Microsecond Molecular Dynamics Simulations with AMBER on GPUs. 2. Explicit Solvent Particle Mesh Ewald. , 2013, Journal of chemical theory and computation.

[49]  Long Chen,et al.  Dynamic load balancing on single- and multi-GPU systems , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[50]  Laxmikant V. Kalé,et al.  Dynamic topology aware load balancing algorithms for molecular dynamics applications , 2009, ICS.

[51]  Steve Plimpton,et al.  Fast parallel algorithms for short-range molecular dynamics , 1993 .

[52]  W. Kohn,et al.  Self-Consistent Equations Including Exchange and Correlation Effects , 1965 .

[53]  Masao Fukushima,et al.  MODYLAS: A Highly Parallelized General-Purpose Molecular Dynamics Simulation Program for Large-Scale Systems with Long-Range Forces Calculated by Fast Multipole Method (FMM) and Highly Scalable Fine-Grained New Parallel Processing Algorithms. , 2013, Journal of chemical theory and computation.

[54]  Timothy C. Germann,et al.  TRILLION-ATOM MOLECULAR DYNAMICS BECOMES A REALITY , 2008 .

[55]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[56]  W. E,et al.  Isotope effects in liquid water via deep potential molecular dynamics , 2019, Molecular Physics.

[57]  Klaus Schulten,et al.  Accelerating Molecular Modeling Applications with GPU Computing , 2009 .

[58]  Rajiv K. Kalia,et al.  Shift-Collapse Acceleration of Generalized Polarizable Reactive Molecular Dynamics for Machine Learning-Assisted Computational Synthesis of Layered Materials , 2018, 2018 IEEE/ACM 9th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (scalA).

[59]  D. Bowler,et al.  O(N) methods in electronic structure calculations. , 2011, Reports on progress in physics. Physical Society.

[60]  Michael M. Resch,et al.  TweTriS: Twenty trillion-atom simulation , 2019, Int. J. High Perform. Comput. Appl..

[61]  P. Hohenberg,et al.  Inhomogeneous Electron Gas , 1964 .

[62]  Weinan E,et al.  Deep neural network for the dielectric response of insulators , 2020 .

[63]  Federico D. Sacerdoti,et al.  Scalable Algorithms for Molecular Dynamics Simulations on Commodity Clusters , 2006, ACM/IEEE SC 2006 Conference (SC'06).

[64]  K. Müller,et al.  Fast and accurate modeling of molecular atomization energies with machine learning. , 2011, Physical review letters.

[65]  Nohad Gresh,et al.  Tinker-HP: a massively parallel molecular dynamics package for multiscale simulations of large complex systems with advanced point dipole polarizable force fields† †Electronic supplementary information (ESI) available. See DOI: 10.1039/c7sc04531j , 2017, Chemical science.

[66]  Laxmikant V. Kalé,et al.  Scalable molecular dynamics with NAMD , 2005, J. Comput. Chem..

[67]  R. Kondor,et al.  Gaussian approximation potentials: the accuracy of quantum mechanics, without the electrons. , 2009, Physical review letters.

[68]  Laxmikant V. Kale,et al.  NAMD2: Greater Scalability for Parallel Molecular Dynamics , 1999 .

[69]  Weile Jia,et al.  Deep Density: circumventing the Kohn-Sham equations via symmetry preserving neural networks , 2019, J. Comput. Phys..

[70]  T. Narumi,et al.  Protein Explorer: A Petaflops Special-Purpose Computer System for Molecular Dynamics Simulations , 2003, ACM/IEEE SC 2003 Conference (SC'03).