Electronic Structure Prediction of Multi-million Atom Systems Through Uncertainty Quantification Enabled Transfer Learning

The ground state electron density -- obtainable using Kohn-Sham Density Functional Theory (KS-DFT) simulations -- contains a wealth of material information, making its prediction via machine learning (ML) models attractive. However, the computational expense of KS-DFT scales cubically with system size which tends to stymie training data generation, making it difficult to develop quantifiably accurate ML models that are applicable across many scales and system configurations. Here, we address this fundamental challenge by employing transfer learning to leverage the multi-scale nature of the training data, while comprehensively sampling system configurations using thermalization. Our ML models are less reliant on heuristics, and being based on Bayesian neural networks, enable uncertainty quantification. We show that our models incur significantly lower data generation costs while allowing confident -- and when verifiable, accurate -- predictions for a wide variety of bulk systems well beyond training, including systems with defects, different alloy compositions, and at unprecedented, multi-million-atom scales. Moreover, such predictions can be carried out using only modest computational resources.

[1]  S. Ong,et al.  Recent advances and outstanding challenges for machine learning interatomic potentials , 2023, Nat. Comput. Sci..

[2]  V. Gavini,et al.  Large-Scale Materials Modeling at Quantum Accuracy: Ab Initio Simulations of Quasicrystals and Interacting Extended Defects in Metallic Alloys , 2023, SC23: International Conference for High Performance Computing, Networking, Storage and Analysis.

[3]  Yuxing Zhou,et al.  Device-scale atomistic modelling of phase-change memory materials , 2023, Nature Electronics.

[4]  Christopher J. Bartel,et al.  CHGNet as a pretrained universal neural network potential for charge-informed atomistic modelling , 2023, Nature Machine Intelligence.

[5]  J. Chelikowsky,et al.  Real-space solution to the electronic structure problem for nearly a million electrons. , 2023, The Journal of chemical physics.

[6]  G. Arora,et al.  Charge-density based evaluation and prediction of stacking fault energies in Ni alloys from DFT and machine learning , 2022, Journal of Applied Physics.

[7]  Wei Hu,et al.  2.5 Million-Atom Ab Initio Electronic-Structure Simulation of Complex Metallic Heterostructures with DGDFT , 2022, SC22: International Conference for High Performance Computing, Networking, Storage and Analysis.

[8]  A. Banerjee,et al.  Solution of the Schrödinger equation for quasi-one-dimensional materials using helical waves , 2022, J. Comput. Phys..

[9]  A. Thompson,et al.  Predicting electronic structures at any length scale with machine learning , 2022, npj Computational Materials.

[10]  A. Thompson,et al.  Predicting the Electronic Structure of Matter on Ultra-Large Scales. , 2022 .

[11]  J. Chelikowsky,et al.  Roadmap on electronic structure codes in the exascale era , 2022, Modelling and Simulation in Materials Science and Engineering.

[12]  Ponkrshnan Thiagarajan,et al.  Jensen-Shannon Divergence Based Novel Loss Functions for Bayesian Neural Networks , 2022, 2209.11366.

[13]  David M. Rogers,et al.  DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization , 2022, Comput. Phys. Commun..

[14]  Kristof T. Schütt,et al.  Roadmap on Machine learning in electronic structure , 2022, Electronic Structure.

[15]  Chi Chen,et al.  A universal graph deep learning interatomic potential for the periodic table , 2022, Nature Computational Science.

[16]  A. Banerjee,et al.  Machine learning based prediction of the electronic structure of quasi-one-dimensional materials under strain , 2022, Physical Review B.

[17]  J. Behler,et al.  A Hessian-based assessment of atomic forces for training machine learning interatomic potentials. , 2021, The Journal of chemical physics.

[18]  A. Bhowmik,et al.  Equivariant graph neural networks for fast electron density estimation of molecules, liquids, and solids , 2021, npj Computational Materials.

[19]  Michele Ceriotti,et al.  Equivariant representations for molecular Hamiltonians and N-center atomic-scale properties. , 2021, The Journal of chemical physics.

[20]  N. Ghosh,et al.  A review on the use of DFT for the prediction of the properties of nanomaterials , 2021, RSC advances.

[21]  Volker L. Deringer,et al.  Gaussian Process Regression for Materials and Molecules , 2021, Chemical reviews.

[22]  M. Ceriotti,et al.  Learning Electron Densities in the Condensed Phase , 2021, Journal of chemical theory and computation.

[23]  Michael Gastegger,et al.  SE(3)-equivariant prediction of molecular wavefunctions and electronic densities , 2021, NeurIPS.

[24]  K. Bhattacharya,et al.  Machine-learned prediction of the electronic fields in a crystal , 2021, Mechanics of Materials.

[25]  A. Banerjee,et al.  Density functional theory method for twisted geometries with application to torsional deformations in group-IV nanotubes , 2021, J. Comput. Phys..

[26]  Gábor Csányi,et al.  Physics-Inspired Structural Representations for Molecules and Materials. , 2021, Chemical reviews.

[27]  Shashank Pathrudkar,et al.  Interpretable machine learning model for the deformation of multiwalled carbon nanotubes , 2021, Physical Review B.

[28]  Pushkar Khairnar,et al.  Explanation and Use of Uncertainty Quantified by Bayesian Neural Network Classifiers for Breast Histopathology Images , 2020, IEEE Transactions on Medical Imaging.

[29]  A. Thompson,et al.  Accelerating Finite-temperature Kohn-Sham Density Functional Theory with Deep Neural Networks , 2020, Physical Review B.

[30]  Clarice D. Aiello,et al.  A Chirality-Based Quantum Leap , 2020, ACS nano.

[31]  A. Banerjee Ab initio framework for systems with helical symmetry: theory, numerical implementation and applications to torsional deformations in nanostructures , 2020, 2008.02267.

[32]  E. Kaxiras,et al.  Electronic-structure methods for twisted moiré layers , 2020, Nature Reviews Materials.

[33]  Jinlong Yang,et al.  High performance computing of DGDFT for tens of thousands of atoms using millions of cores on Sunway TaihuLight. , 2020, Science bulletin.

[34]  Andrew J. Medford,et al.  SPARC: Simulation Package for Ab-initio Real-space Calculations , 2020, SoftwareX.

[35]  Lin Lin,et al.  Pushing the Limit of Molecular Dynamics with Ab Initio Accuracy to 100 Million Atoms with Machine Learning , 2020, SC20: International Conference for High Performance Computing, Networking, Storage and Analysis.

[36]  Phanish Suryanarayana,et al.  M-SPARC: Matlab-Simulation Package for Ab-initio Real-space Calculations , 2019, SoftwareX.

[37]  Weile Jia,et al.  Deep Density: circumventing the Kohn-Sham equations via symmetry preserving neural networks , 2019, J. Comput. Phys..

[38]  Hui Xiong,et al.  A Comprehensive Survey on Transfer Learning , 2019, Proceedings of the IEEE.

[39]  Kipton Barros,et al.  Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning , 2019, Nature Communications.

[40]  G. R. Schleder,et al.  From DFT to machine learning: recent approaches to materials science–a review , 2019, Journal of Physics: Materials.

[41]  Phanish Suryanarayana,et al.  Symmetry-adapted real-space density functional theory for cylindrical geometries: Application to large group-IV nanotubes , 2019, Physical Review B.

[42]  Filippo Federici Canova,et al.  DScribe: Library of Descriptors for Machine Learning in Materials Science , 2019, Comput. Phys. Commun..

[43]  Anand Chandrasekaran,et al.  Solving the electronic structure problem with machine learning , 2019, npj Computational Materials.

[44]  Andrew Gordon Wilson,et al.  Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning , 2019, ICLR.

[45]  Y. Ikeda,et al.  Ab initio phase stabilities and mechanical properties of multicomponent alloys: A comprehensive review for high entropy alloys and compositionally complex alloys , 2019, Materials Characterization.

[46]  C. Pickard,et al.  Managing uncertainty in data-derived densities to accelerate density functional theory , 2018, Journal of Physics: Materials.

[47]  M. Ceriotti,et al.  A Transferable Machine-Learning Model of the Electron Density , 2018, 1809.05349.

[48]  John M. Alred,et al.  Machine learning electron density in sulfur crosslinked carbon nanotubes , 2018, Composites Science and Technology.

[49]  Gus L. W. Hart,et al.  Accelerating high-throughput searches for new alloys with active learning of interatomic potentials , 2018, Computational Materials Science.

[50]  Jörg Behler,et al.  Automatic selection of atomic fingerprints and reference configurations for machine-learning potentials. , 2018, The Journal of chemical physics.

[51]  Li Li,et al.  Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds , 2018, ArXiv.

[52]  M Gastegger,et al.  wACSF-Weighted atom-centered symmetry functions as descriptors in machine learning potentials. , 2017, The Journal of chemical physics.

[53]  Phanish Suryanarayana,et al.  Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations. , 2017, Journal of chemical theory and computation.

[54]  E Weinan,et al.  DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics , 2017, Comput. Phys. Commun..

[55]  Gábor Csányi,et al.  Machine learning unifies the modeling of materials and molecules , 2017, Science Advances.

[56]  Andrea Grisafi,et al.  Symmetry-Adapted Machine Learning for Tensorial Properties of Atomistic Systems. , 2017, Physical review letters.

[57]  Phanish Suryanarayana,et al.  SQDFT: Spectral Quadrature method for large-scale parallel O(N) Kohn-Sham calculations at high temperature , 2017, Comput. Phys. Commun..

[58]  Matthias Rupp,et al.  Unified representation of molecules and crystals for machine learning , 2017, Mach. Learn. Sci. Technol..

[59]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[60]  Wei Chen,et al.  Predicting defect behavior in B2 intermetallics by merging ab initio modeling and machine learning , 2016, npj Computational Materials.

[61]  Rampi Ramprasad,et al.  Machine Learning Force Fields: Construction, Validation, and Outlook , 2016, 1610.02098.

[62]  Li Li,et al.  Bypassing the Kohn-Sham equations with machine learning , 2016, Nature Communications.

[63]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Friedhelm Bechstedt,et al.  Many-Body Approach to Electronic Excitations: Concepts and Applications , 2016 .

[65]  Kevin Gimpel,et al.  Gaussian Error Linear Units (GELUs) , 2016, 1606.08415.

[66]  Chao Yang,et al.  Chebyshev polynomial filtered subspace iteration in the discontinuous Galerkin method for large-scale electronic structure calculations. , 2016, The Journal of chemical physics.

[67]  Phanish Suryanarayana,et al.  SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Isolated clusters , 2016, Comput. Phys. Commun..

[68]  Phanish Suryanarayana,et al.  Spectral Quadrature method for accurate O(N) electronic structure calculations of metals and insulators , 2016, Comput. Phys. Commun..

[69]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[70]  Gábor Csányi,et al.  Comparing molecules and solids across structural and alchemical space. , 2015, Physical chemistry chemical physics : PCCP.

[71]  Phanish Suryanarayana,et al.  Periodic Pulay method for robust and efficient convergence acceleration of self-consistent field iterations , 2015, 1512.01604.

[72]  Pierre Hirel,et al.  Atomsk: A tool for manipulating and converting atomic data files , 2015, Comput. Phys. Commun..

[73]  Julien Cornebise,et al.  Weight Uncertainty in Neural Network , 2015, ICML.

[74]  Peter V. Gehler,et al.  Pattern Recognition , 2015, Lecture Notes in Computer Science.

[75]  Ryan P. Adams,et al.  Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks , 2015, ICML.

[76]  Stefan Goedecker,et al.  Accurate and efficient linear scaling DFT calculations with universal applicability. , 2015, Physical chemistry chemical physics : PCCP.

[77]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[78]  Christian Trott,et al.  Spectral neighbor analysis method for automated generation of quantum-accurate interatomic potentials , 2014, J. Comput. Phys..

[79]  Jien-Wei Yeh,et al.  High-Entropy Alloys , 2014 .

[80]  Phani Motamarri,et al.  Subquadratic-scaling subspace projection method for large-scale Kohn-Sham density functional theory calculations using spectral finite-element discretization , 2014, 1406.2600.

[81]  Atsuto Seko,et al.  Sparse representation for a potential energy surface , 2014, 1403.7995.

[82]  T. van Mourik,et al.  Density functional theory across chemistry, physics and biology , 2014, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[83]  R. Gorbachev Van der Waals heterostructures , 2014, Nature Reviews Methods Primers.

[84]  Chao Yang,et al.  SIESTA-PEXSI: massively parallel method for efficient and accurate ab initio materials simulation without matrix diagonalization , 2014, Journal of physics. Condensed matter : an Institute of Physics journal.

[85]  Tianqi Chen,et al.  Stochastic Gradient Hamiltonian Monte Carlo , 2014, ICML.

[86]  D. Hamann Optimized norm-conserving Vanderbilt pseudopotentials , 2013, 1306.4707.

[87]  R. Kondor,et al.  On representing chemical environments , 2012, 1209.3140.

[88]  Chao Yang,et al.  Accelerating atomic orbital-based electronic structure calculation via pole expansion and selected inversion , 2012, Journal of physics. Condensed matter : an Institute of Physics journal.

[89]  Klaus-Robert Müller,et al.  Finding Density Functionals with Machine Learning , 2011, Physical review letters.

[90]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[91]  K. Müller,et al.  Fast and accurate modeling of molecular atomization energies with machine learning. , 2011, Physical review letters.

[92]  Yee Whye Teh,et al.  Bayesian Learning via Stochastic Gradient Langevin Dynamics , 2011, ICML.

[93]  D. Miracle,et al.  Mechanical properties of Nb25Mo25Ta25W25 and V20Nb20Mo20Ta20W20 refractory high entropy alloys , 2011 .

[94]  Lexing Ying,et al.  SelInv---An Algorithm for Selected Inversion of a Sparse Symmetric Matrix , 2011, TOMS.

[95]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[96]  P. Liaw,et al.  Refractory high-entropy alloys , 2010 .

[97]  W. Maley Literature , 2010 .

[98]  E. Weinan,et al.  Fast algorithm for extracting the diagonal of the inverse matrix with application to the electronic structure analysis of metallic systems , 2009 .

[99]  D. Bailey,et al.  The linearly scaling 3D fragment method for large scale electronic structure calculations , 2009 .

[100]  M. Ortiz,et al.  Vacancy clustering and prismatic dislocation loop formation in aluminum , 2007 .

[101]  Michele Parrinello,et al.  Generalized neural-network representation of high-dimensional potential-energy surfaces. , 2007, Physical review letters.

[102]  Gerbrand Ceder,et al.  Toward Computational Materials Design: The Impact of Density Functional Theory on Materials Research , 2006 .

[103]  D. Nguyen-Manh,et al.  Electronic structure of complex Hume-Rothery phases and quasicrystals in transition metal aluminides , 2005 .

[104]  W. Kohn,et al.  Nearsightedness of electronic matter. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[105]  Chris-Kriton Skylaris,et al.  Introducing ONETEP: linear-scaling density functional simulations on parallel computers. , 2005, The Journal of chemical physics.

[106]  M C Payne,et al.  "Learn on the fly": a hybrid classical and quantum-mechanical molecular dynamics simulation. , 2004, Physical review letters.

[107]  S. Datta Quantum Transport: Atom to Transistor , 2004 .

[108]  S. Raju,et al.  The high temperature bulk modulus of aluminium: an assessment using experimental enthalpy and thermal expansion data , 2002 .

[109]  C. Woodward,et al.  Flexible Ab initio boundary conditions: simulating isolated dislocations in bcc Mo and Ta. , 2002, Physical review letters.

[110]  David R. Bowler,et al.  Recent progress in linear scaling ab initio electronic structure techniques , 2002 .

[111]  S. F. Fischer,et al.  Critical magnetic properties of disordered polycrystalline Cr75Fe25 and Cr70Fe30 alloys , 2002 .

[112]  S. Goedecker Linear scaling electronic structure methods , 1999 .

[113]  D. Sánchez-Portal,et al.  LINEAR-SCALING AB-INITIO CALCULATIONS FOR LARGE AND COMPLEX SYSTEMS , 1999, cond-mat/9904159.

[114]  R. Resta,et al.  Electron Localization in the Insulating State , 1998, cond-mat/9808151.

[115]  Huaiyu Zhu On Information and Sufficiency , 1997 .

[116]  Burke,et al.  Generalized Gradient Approximation Made Simple. , 1996, Physical review letters.

[117]  A. C. Stowe,et al.  Learning from Literature , 1996 .

[118]  Kohn,et al.  Density functional and density matrix method scaling linearly with the number of atoms. , 1996, Physical review letters.

[119]  Weitao Yang,et al.  A density‐matrix divide‐and‐conquer approach for electronic structure calculations of large molecules , 1995 .

[120]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[121]  Ferreira,et al.  Electronic properties of random alloys: Special quasirandom structures. , 1990, Physical review. B, Condensed matter.

[122]  Foulkes,et al.  Tight-binding models and density-functional theory. , 1989, Physical review. B, Condensed matter.

[123]  Denis J. Evans,et al.  The Nose–Hoover thermostat , 1985 .

[124]  M. Jaroš,et al.  Electronic properties of semiconductor alloy systems , 1985 .

[125]  Harris Simplified method for calculating the energy of weakly interacting fragments. , 1985, Physical review. B, Condensed matter.

[126]  Joseph Callaway,et al.  Inhomogeneous Electron Gas , 1973 .

[127]  W. Kohn,et al.  Self-Consistent Equations Including Exchange and Correlation Effects , 1965 .

[128]  A. S. Cooper Precise lattice constants of germanium, aluminum, gallium arsenide, uranium, sulphur, quartz and sapphire , 1962 .

[129]  James B. Bean,et al.  Aluminum , 1867, The American journal of dental science.

[130]  P. Dupont,et al.  An Importance Weighted Feature Selection Stability Measure , 2021, J. Mach. Learn. Res..

[131]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[132]  C. Bolliger,et al.  Linear Scaling Electronic Structure Methods , 2008 .

[133]  Kevin Leung,et al.  Designing meaningful density functional theory calculations in materials science—a primer , 2004 .

[134]  R. Martin,et al.  Electronic Structure: Basic Theory and Practical Methods , 2004 .

[135]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[136]  J. Hafner Atomic-scale computational materials science ☆ , 2000 .

[137]  Charles M. Bishop,et al.  Ensemble learning in Bayesian neural networks , 1998 .

[138]  Rodrigo Freitas,et al.  Early Career Materials Researcher Prospective Machine‑learning potentials for crystal defects , 2022 .