DGDFT: A massively parallel method for large scale density functional theory calculations.

We describe a massively parallel implementation of the recently developed discontinuous Galerkin density functional theory (DGDFT) method, for efficient large-scale Kohn-Sham DFT based electronic structure calculations. The DGDFT method uses adaptive local basis (ALB) functions generated on-the-fly during the self-consistent field iteration to represent the solution to the Kohn-Sham equations. The use of the ALB set provides a systematic way to improve the accuracy of the approximation. By using the pole expansion and selected inversion technique to compute electron density, energy, and atomic forces, we can make the computational complexity of DGDFT scale at most quadratically with respect to the number of electrons for both insulating and metallic systems. We show that for the two-dimensional (2D) phosphorene systems studied here, using 37 basis functions per atom allows us to reach an accuracy level of 1.3 × 10(-4) Hartree/atom in terms of the error of energy and 6.2 × 10(-4) Hartree/bohr in terms of the error of atomic force, respectively. DGDFT can achieve 80% parallel efficiency on 128,000 high performance computing cores when it is used to study the electronic structure of 2D phosphorene systems with 3500-14 000 atoms. This high parallel efficiency results from a two-level parallelization scheme that we will describe in detail.

[1]  Chris-Kriton Skylaris,et al.  Introducing ONETEP: linear-scaling density functional simulations on parallel computers. , 2005, The Journal of chemical physics.

[2]  Joost VandeVondele,et al.  Linear Scaling Self-Consistent Field Calculations with Millions of Atoms in the Condensed Phase. , 2012, Journal of chemical theory and computation.

[3]  Xianfan Xu,et al.  Phosphorene: an unexplored 2D semiconductor with a high hole mobility. , 2014, ACS nano.

[4]  Taisuke Ozaki,et al.  Efficient projector expansion for the ab initio LCAO method , 2005 .

[5]  L. Trefethen.  Spectral Methods in MATLAB , 2000 .

[6]  D R Bowler,et al.  Calculations for millions of atoms with density functional theory: linear scaling shows its potential , 2009, Journal of physics. Condensed matter : an Institute of Physics journal.

[7]  Chao Yang,et al.  Electronic structure and aromaticity of large-scale hexagonal graphene nanoflakes. , 2014, The Journal of chemical physics.

[8]  Wei Hu,et al.  Electronic Structures of Hybrid Graphene/Phosphorene Nanocomposite , 2014, 1411.1781.

[9]  Teter,et al.  Separable dual-space Gaussian pseudopotentials. , 1996, Physical review. B, Condensed matter.

[10]  Likai Li,et al.  Black phosphorus field-effect transistors. , 2014, Nature nanotechnology.

[11]  S. Goedecker.  Linear scaling electronic structure methods , 1999 .

[12]  Emanuel H. Rubensson,et al.  Bringing about matrix sparsity in linear‐scaling electronic structure calculations , 2011, J. Comput. Chem..

[13]  E Weinan,et al.  Optimized local basis set for Kohn-Sham density functional theory , 2011, J. Comput. Phys..

[14]  S. Goedecker,et al.  Relativistic separable dual-space Gaussian pseudopotentials from H to Rn , 1998, cond-mat/9803286.

[15]  W. Kohn,et al.  Self-Consistent Equations Including Exchange and Correlation Effects , 1965 .

[16]  Leonard Kleinman,et al.  Efficacious Form for Model Pseudopotentials , 1982 .

[17]  Kresse,et al.  Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. , 1996, Physical review. B, Condensed matter.

[18]  Andrew V. Knyazev,et al.  Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method , 2001, SIAM J. Sci. Comput..

[19]  Stefan Goedecker,et al.  Daubechies wavelets for linear scaling density functional theory. , 2014, The Journal of chemical physics.

[20]  Michele Parrinello,et al.  Quickstep: Fast and accurate density functional calculations using a mixed Gaussian and plane waves approach , 2005, Comput. Phys. Commun..

[21]  Chao Yang,et al.  A posteriori error estimator for adaptive local basis functions to solve Kohn-Sham density functional theory , 2014, 1401.0920.

[22]  Burke,et al.  Generalized Gradient Approximation Made Simple. , 1996, Physical review letters.

[23]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[24]  Douglas N. Arnold,et al.  Unified Analysis of Discontinuous Galerkin Methods for Elliptic Problems , 2001, SIAM J. Numer. Anal..

[25]  D. Bowler,et al.  O(N) methods in electronic structure calculations. , 2011, Reports on progress in physics. Physical Society.

[26]  Stefan Goedecker,et al.  ABINIT: First-principles approach to material and nanosystem properties , 2009, Comput. Phys. Commun..

[27]  Yihan Shao,et al.  Sparse matrix multiplications for linear scaling electronic structure calculations in an atom‐centered basis set using multiatom blocks , 2003, J. Comput. Chem..

[28]  Xinming Qin,et al.  HONPAS: A linear scaling open‐source solution for large system simulations , 2015 .

[29]  M. J. Gillan,et al.  Order-N first-principles calculations with the conquest code , 2007, Comput. Phys. Commun..

[30]  Zhenyu Li,et al.  Linear scaling electronic structure calculations with numerical atomic basis set , 2010 .

[31]  Wei Ji,et al.  High-mobility transport anisotropy and linear dichroism in few-layer black phosphorus , 2014, Nature communications.

[32]  Chao Yang,et al.  Accelerating atomic orbital-based electronic structure calculation via pole expansion and selected inversion , 2012, Journal of physics. Condensed matter : an Institute of Physics journal.

[33]  F. Guinea,et al.  The electronic properties of graphene , 2007, Reviews of Modern Physics.

[34]  P. Pulay.  Convergence acceleration of iterative sequences. the case of scf iteration , 1980 .

[35]  Joost VandeVondele,et al.  Sparse matrix multiplication: The distributed block-compressed sparse row library , 2014, Parallel Comput..

[36]  T. Arias,et al.  Iterative minimization techniques for ab initio total energy calculations: molecular dynamics and co , 1992 .

[37]  Jun Dai,et al.  Bilayer Phosphorene: Effect of Stacking Order on Bandgap and Its Potential Applications in Thin-Film Solar Cells. , 2014, The journal of physical chemistry letters.

[38]  Andre K. Geim,et al.  Electric Field Effect in Atomically Thin Carbon Films , 2004, Science.

[39]  Stefano de Gironcoli,et al.  QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials , 2009, Journal of physics. Condensed matter : an Institute of Physics journal.

[40]  D. Arnold.  An Interior Penalty Finite Element Method with Discontinuous Elements , 1982 .

[41]  Chao Yang,et al.  SIESTA-PEXSI: massively parallel method for efficient and accurate ab initio materials simulation without matrix diagonalization , 2014, Journal of physics. Condensed matter : an Institute of Physics journal.

[42]  P. Hohenberg,et al.  Inhomogeneous Electron Gas , 1964 .

[43]  Eric Shea-Brown,et al.  Reliability of Layered Neural Oscillator Networks , 2008 .

[44]  D. Sánchez-Portal,et al.  The SIESTA method for ab initio order-N materials simulation , 2001, cond-mat/0111138.

[45]  E Weinan,et al.  Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation , 2011, J. Comput. Phys..

[46]  Xianfan Xu,et al.  Phosphorene: an unexplored 2D semiconductor with a high hole mobility. , 2014, ACS nano.

[47]  N. Mermin.  Thermal Properties of the Inhomogeneous Electron Gas , 1965 .

[48]  Patrick R. Briddon,et al.  Highly efficient method for Kohn-Sham density functional calculations of 500–10 000 atom systems , 2009 .

[49]  Chao Yang,et al.  Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework II: Force, vibration, and molecular dynamics calculations , 2012, J. Comput. Phys..

[50]  Chao Yang,et al.  Electronic Structure of Large-Scale Graphene Nanoflakes , 2014, 1409.1211.

[51]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[52]  G. Kresse,et al.  Ab initio molecular dynamics for liquid metals. , 1993 .

[53]  Wei Hu,et al.  Tunable Schottky contacts in hybrid graphene–phosphorene nanocomposites , 2014, 1411.1781.

[54]  Andre K. Geim,et al.  The rise of graphene. , 2007, Nature materials.

[55]  Lukas Krämer,et al.  Parallel solution of partial symmetric eigenvalue problems from electronic structure calculations , 2011, Parallel Comput..

[56]  W. Marsden.  I and J , 2012 .

[57]  François Gygi,et al.  Architecture of Qbox: A scalable first-principles molecular dynamics code , 2008, IBM J. Res. Dev..