Quantum Dynamics at Scale: Ultrafast Control of Emergent Functional Materials

Confluence of extreme-scale quantum dynamics simulations (i.e. quantum@scale) and cutting-edge x-ray free-electron laser experiments are revolutionizing materials science. An archetypal example is the exciting concept of using picosecond light pulses to control emergent material properties on demand in atomically-thin layered materials. This paper describes efforts to scale our quantum molecular dynamics engine toward the United States' first exaflop/s computer, under an Aurora Early Science Program project named "Metascalable layered material genome". Key algorithmic and computing techniques incorporated are: (1) globally-scalable and locally-fast solvers within a linear-scaling divide-conquer-recombine algorithmic framework; (2) algebraic 'BLASification' of computational kernels; and (3) data alignment and loop restructuring, along with register and cache blocking, for enhanced vectorization and efficient memory access. The resulting weak-scaling parallel efficiency was 0.93 on 131,072 Intel Xeon Phi cores for a 56.6 million atom (or 169 million valence-electron) system, whereas the various code transformations achieved 5-fold speedup. The optimized simulation engine allowed us for the first time to establish a significant effect of substrate on the dynamics of layered material upon electronic excitation.

[1]  Shogo Fukushima,et al.  QXMD: An open-source program for nonadiabatic quantum molecular dynamics , 2019, SoftwareX.

[2]  S. Goedecker.  Linear scaling electronic structure methods , 1999 .

[3]  Yen-Chen Liu,et al.  Knights Landing: Second-Generation Intel Xeon Phi Product , 2016, IEEE Micro.

[4]  Prashant Shekhar,et al.  Heterogeneous to homogeneous melting transition visualized with ultrafast electron diffraction , 2018, Science.

[5]  W. Kohn,et al.  Self-Consistent Equations Including Exchange and Correlation Effects , 1965 .

[6]  D. Roweth,et al.  Cray XC ® Series Network , 2012 .

[7]  Rajiv K. Kalia,et al.  A scalable parallel algorithm for dynamic range-limited n-tuple computation in many-body molecular dynamics simulation , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[8]  D. Bowler,et al.  O(N) methods in electronic structure calculations. , 2011, Reports on progress in physics. Physical Society.

[9]  P. Hohenberg,et al.  Inhomogeneous Electron Gas , 1964 .

[10]  Supa Hannongbua,et al.  Shift/collapse on neighbor list (SC-NBL): Fast evaluation of dynamic many-body potentials in molecular dynamics simulations , 2019, Comput. Phys. Commun..

[11]  Paul Messina,et al.  Quantum Molecular Dynamics in the Post-Petaflops Era , 2015, Computer.

[12]  Robert F Service.  Design for U.S. exascale computer takes shape. , 2018, Science.

[13]  Kohn,et al.  Density functional and density matrix method scaling linearly with the number of atoms. , 1996, Physical review letters.

[14]  Car,et al.  Unified approach for molecular dynamics and density-functional theory. , 1985, Physical review letters.

[15]  W. Kohn,et al.  Nearsightedness of electronic matter. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[16]  P. Ajayan,et al.  Optical Control of Non-Equilibrium Phonon Dynamics. , 2019, Nano letters.

[17]  Phanish Suryanarayana,et al.  Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations. , 2017, Journal of chemical theory and computation.

[18]  E Weinan,et al.  Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation , 2011, J. Comput. Phys..

[19]  Priya Vashishta,et al.  A divide-conquer-recombine algorithmic paradigm for large spatiotemporal quantum molecular dynamics simulations. , 2014, The Journal of chemical physics.

[20]  Yang,et al.  Direct calculation of electron density in density-functional theory. , 1991, Physical review letters.

[21]  Paul Messina,et al.  Metascalable Quantum Molecular Dynamics Simulations of Hydrogen-on-Demand , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[22]  D. Brandt,et al.  Multi-level adaptive solutions to boundary-value problems math comptr , 1977 .

[23]  Rajiv K. Kalia,et al.  Embedded divide-and-conquer algorithm on hierarchical real-space grids: parallel molecular dynamics simulation based on linear-scaling density functional theory , 2005, Comput. Phys. Commun..

[24]  Chao Yang,et al.  DGDFT: A massively parallel method for large scale density functional theory calculations. , 2015, The Journal of chemical physics.

[25]  Rajiv K. Kalia,et al.  Shift-Collapse Acceleration of Generalized Polarizable Reactive Molecular Dynamics for Machine Learning-Assisted Computational Synthesis of Layered Materials , 2018, 2018 IEEE/ACM 9th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (scalA).

[26]  Nick Woods,et al.  Computing the self-consistent field in Kohn–Sham density functional theory , 2019, Journal of physics. Condensed matter : an Institute of Physics journal.

[27]  R. Ramesh,et al.  Observation of room-temperature polar skyrmions , 2019, Nature.

[28]  Anisotropic structural dynamics of monolayer crystals revealed by femtosecond surface X-ray scattering , 2018, Nature Photonics.

[29]  R. Gorbachev.  Van der Waals heterostructures , 2014, Nature Reviews Methods Primers.

[30]  R. Averitt,et al.  Towards properties on demand in quantum materials. , 2017, Nature materials.

[31]  Sébastien Boutet,et al.  Linac Coherent Light Source: The first five years , 2016 .

[32]  Robert B. Ross,et al.  Watch Out for the Bully! Job Interference Study on Dragonfly Network , 2016, SC16: International Conference for High Performance Computing, Networking, Storage and Analysis.

[33]  Chun Chen,et al.  Hierarchical parallelization and optimization of high-order stencil computations on multicore clusters , 2012, The Journal of Supercomputing.

[34]  Wei Ruan,et al.  Giant bandgap renormalization and excitonic effects in a monolayer transition metal dichalcogenide semiconductor. , 2014, Nature materials.

[35]  A. Nakano,et al.  Divide-and-conquer density functional theory on hierarchical real-space grids: Parallel implementation and applications , 2008 .

[36]  T. Arias,et al.  Iterative minimization techniques for ab initio total energy calculations: molecular dynamics and co , 1992 .

[37]  A. Nakano,et al.  Semiconductor-metal structural phase transformation in MoTe2 monolayers by electronic excitation. , 2018, Nanoscale.

[38]  Takashi Taniguchi,et al.  Unconventional superconductivity in magic-angle graphene superlattices , 2018, Nature.

[39]  V. Morozov,et al.  Early Evaluation of the Cray XC 40 Xeon Phi System ‘ Theta ’ at Argonne , 2017 .

[40]  P. Ajayan,et al.  Ultrafast non-radiative dynamics of atomically thin MoSe2 , 2017, Nature Communications.

[41]  Weiqiang Wang,et al.  A metascalable computing framework for large spatiotemporal-scale atomistic simulations , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[42]  Priya Vashishta,et al.  Electronic Origin of Optically-Induced Sub-Picosecond Lattice Dynamics in MoSe2 Monolayer. , 2018, Nano letters.