Massively Parallel Implementation of Explicitly Correlated Coupled-Cluster Singles and Doubles Using TiledArray Framework.

A new distributed-memory massively parallel implementation of standard and explicitly correlated (F12) coupled-cluster singles and doubles (CCSD) with canonical O(N6) computational complexity is described. The implementation is based on the TiledArray tensor framework. Novel features of the implementation include (a) all data greater than O(N) is distributed in memory and (b) the mixed use of density fitting and integral-driven formulations that optionally allows to avoid storage of tensors with three and four unoccupied indices. Excellent strong scaling is demonstrated on a multicore shared-memory computer, a commodity distributed-memory computer, and a national-scale supercomputer. The performance on a shared-memory computer is competitive with the popular CCSD implementations in ORCA and Psi4. Moreover, the CCSD performance on a commodity-size cluster significantly improves on the state-of-the-art package NWChem. The large-scale parallel explicitly correlated coupled-cluster implementation makes routine accurate estimation of the coupled-cluster basis set limit for molecules with 20 or more atoms. Thus, it can provide valuable benchmarks for the merging reduced-scaling coupled-cluster approaches. The new implementation allowed us to revisit the basis set limit for the CCSD contribution to the binding energy of π-stacked uracil dimer, a challenging paradigm of π-stacking interactions from the S66 benchmark database. The revised value for the CCSD correlation binding energy obtained with the help of quadruple-ζ CCSD computations, -8.30 ± 0.02 kcal/mol, is significantly different from the S66 reference value, -8.50 kcal/mol, as well as other CBS limit estimates in the recent literature.

[1]  Robert A. van de Geijn,et al.  SUMMA: Scalable Universal Matrix Multiplication Algorithm , 1995 .

[2]  Peter Pulay,et al.  Parallel Calculation of Coupled Cluster Singles and Doubles Wave Functions Using Array Files. , 2007, Journal of chemical theory and computation.

[3]  Beverly A. Sanders,et al.  A Block-Oriented Language and Runtime System for Tensor Algebra with Very Large Arrays , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[4]  Sriram Krishnamoorthy,et al.  Scalable implementations of accurate excited-state coupled cluster theories: Application of high-level methods to porphyrin-based systems , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[5]  R. T. Pack,et al.  Cusp Conditions for Molecular Wavefunctions , 1966 .

[6]  J. Noga,et al.  Alternative formulation of the matrix elements in MP2‐R12 theory , 2005 .

[7]  Curtis L. Janssen,et al.  An efficient reformulation of the closed‐shell coupled cluster single and double excitation (CCSD) equations , 1988 .

[8]  Justus A. Calvin,et al.  Scalable task-based algorithm for multiplication of block-rank-sparse matrices , 2015, IA3@SC.

[9]  Matthew L. Leininger,et al.  Psi4: an open‐source ab initio electronic structure program , 2012 .

[10]  Hans-Joachim Werner,et al.  Scalable electron correlation methods I.: PNO-LMP2 with linear scaling in the molecular size and near-inverse-linear scaling in the number of processors. , 2015, Journal of chemical theory and computation.

[11]  Rodney J Bartlett,et al.  Parallel implementation of the equation-of-motion coupled-cluster singles and doubles method and application for radical adducts of cytosine. , 2009, The Journal of chemical physics.

[12]  Edward F. Valeev,et al.  Prediction of Reaction Barriers and Thermochemical Properties with Explicitly Correlated Coupled-Cluster Methods: A Basis Set Assessment. , 2012, Journal of chemical theory and computation.

[13]  F. Coester,et al.  Bound states of a many-particle system , 1958 .

[14]  So Hirata,et al.  Explicitly correlated coupled-cluster singles and doubles method based on complete diagrammatic equations. , 2008, The Journal of chemical physics.

[15]  Hans-Joachim Werner,et al.  Extrapolating MP2 and CCSD explicitly correlated correlation energies to the complete basis set limit with first and second row correlation consistent basis sets. , 2009, The Journal of chemical physics.

[16]  Sriram Krishnamoorthy,et al.  Active-space completely-renormalized equation-of-motion coupled-cluster formalism: Excited-state studies of green fluorescent protein, free-base porphyrin, and oligoporphyrin dimer. , 2010, The Journal of chemical physics.

[17]  D. Tew,et al.  Communications: Accurate and efficient approximations to explicitly correlated coupled-cluster singles and doubles, CCSD-F12. , 2010, The Journal of chemical physics.

[18]  Christof Hättig,et al.  Quintuple-ζ quality coupled-cluster correlation energies with triple-ζ basis sets , 2007 .

[19]  Tomasz Janowski,et al.  Efficient Parallel Implementation of the CCSD External Exchange Operator and the Perturbative Triples (T) Energy Calculation. , 2008, Journal of chemical theory and computation.

[20]  Peter Pulay,et al.  Local configuration interaction: An efficient approach for larger molecules , 1985 .

[21]  M. Head‐Gordon,et al.  A fifth-order perturbation comparison of electron correlation theories , 1989 .

[22]  Frank Neese,et al.  The ORCA program system , 2012 .

[23]  S. Hirata.  Tensor Contraction Engine: Abstraction and Automated Parallel Implementation of Configuration-Interaction, Coupled-Cluster, and Many-Body Perturbation Theories , 2003 .

[24]  Jan Almlöf,et al.  Elimination of energy denominators in Møller—Plesset perturbation theory by a Laplace transform approach , 1991 .

[25]  Edward F. Valeev,et al.  Explicitly correlated combined coupled-cluster and perturbation methods. , 2009, The Journal of chemical physics.

[26]  Alán Aspuru-Guzik,et al.  Advances in molecular quantum chemistry contained in the Q-Chem 4 program package , 2014, Molecular Physics.

[27]  Tjerk P. Straatsma,et al.  NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations , 2010, Comput. Phys. Commun..

[28]  W. Klopper,et al.  Coupled-cluster theory with simplified linear-r(12) corrections: the CCSD(R12) model. , 2005, The Journal of chemical physics.

[29]  Angela K. Wilson,et al.  Møller-Plesset correlation energies in a localized orbital basis using a Laplace transform technique , 1997 .

[30]  Beverly A. Sanders,et al.  The Super Instruction Architecture: A Framework for High-Productivity Parallel Implementation of Coupled-Cluster Methods on Petascale Computers , 2011 .

[31]  Pavel Hobza,et al.  S66: A Well-balanced Database of Benchmark Interaction Energies Relevant to Biomolecular Structures , 2011, Journal of chemical theory and computation.

[32]  Frank Neese,et al.  Geminal-spanning orbitals make explicitly correlated reduced-scaling coupled-cluster methods robust, yet simple. , 2014, The Journal of chemical physics.

[33]  D. Tew,et al.  Implementation of the full explicitly correlated coupled-cluster singles and doubles model CCSD-F12 with optimally reduced auxiliary basis dependence. , 2008, The Journal of chemical physics.

[34]  Hans-Joachim Werner,et al.  Scalable Electron Correlation Methods. 2. Parallel PNO-LMP2-F12 with Near Linear Scaling in the Molecular Size. , 2015, Journal of chemical theory and computation.

[35]  J. Noga,et al.  Coupled cluster theory that takes care of the correlation cusp by inclusion of linear terms in the interelectronic coordinates , 1994 .

[36]  So Hirata,et al.  Higher-order explicitly correlated coupled-cluster methods. , 2009, The Journal of chemical physics.

[37]  Philippe Y. Ayala,et al.  Linear scaling coupled cluster and perturbation theories in the atomic orbital basis , 1999 .

[38]  F. Coester,et al.  Short-range correlations in nuclear wave functions , 1960 .

[39]  Andrey Asadchev,et al.  Fast and Flexible Coupled Cluster Implementation. , 2013, Journal of chemical theory and computation.

[40]  Gregory Beylkin,et al.  Multiresolution quantum chemistry: basic theory and initial applications. , 2004, The Journal of chemical physics.

[41]  Alistair P. Rendell,et al.  A direct coupled cluster algorithm for massively parallel computers , 1997 .

[42]  Kalyana Chadalavada,et al.  Optimization of the Coupled Cluster Implementation in NWChem on Petascale Parallel Architectures. , 2014, Journal of chemical theory and computation.

[43]  James Demmel,et al.  Cyclops Tensor Framework: Reducing Communication and Eliminating Load Imbalance in Massively Parallel Contractions , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[44]  R. Bartlett,et al.  Coupled-cluster theory in quantum chemistry , 2007 .

[45]  Manoj K. Kesharwani,et al.  The S66x8 benchmark for noncovalent interactions revisited: explicitly correlated ab initio methods and density functional theory. , 2016, Physical chemistry chemical physics : PCCP.

[46]  Frank Neese,et al.  SparseMaps--A systematic infrastructure for reduced-scaling electronic structure methods. III. Linear-scaling multireference domain-based pair natural orbital N-electron valence perturbation theory. , 2016, The Journal of chemical physics.

[47]  David E. Bernholdt,et al.  Automatic code generation for many-body electronic structure methods: the tensor contraction engine , 2006 .

[48]  Wim Klopper,et al.  CC-R12, a correlation cusp corrected coupled-cluster method with a pilot application to the Be2 potential curve , 1992 .

[49]  M. Krauss,et al.  Configuration‐Interaction Calculation of H3 and H2 , 1965 .

[50]  Alistair P. Rendell,et al.  Quantum chemistry on parallel computer architectures: coupled-cluster theory applied to the bending potential of fulminic acid , 1992 .

[51]  Hans-Joachim Werner,et al.  A simple and efficient CCSD(T)-F12 approximation. , 2007, The Journal of chemical physics.

[52]  Hans-Joachim Werner,et al.  Systematically convergent basis sets for explicitly correlated wavefunctions: the atoms H, He, B-Ne, and Al-Ar. , 2008, The Journal of chemical physics.

[53]  Dimitrios G Liakos,et al.  Efficient and accurate approximations to the local coupled cluster singles doubles method using a truncated pair natural orbital basis. , 2009, The Journal of chemical physics.

[54]  Seiichiro Ten-no,et al.  Explicitly correlated second order perturbation theory: introduction of a rational generator and numerical quadratures. , 2004, The Journal of chemical physics.

[55]  Kevin E. Riley,et al.  Extensions of the S66 Data Set: More Accurate Interaction Energies and Angular-Displaced Nonequilibrium Geometries , 2011 .

[56]  Edward F. Valeev,et al.  Coupled-cluster methods with perturbative inclusion of explicitly correlated terms: a preliminary investigation. , 2008, Physical chemistry chemical physics : PCCP.

[57]  K. Kitaura,et al.  Fragment molecular orbital method: an approximate computational method for large molecules , 1999 .

[58]  Hans-Joachim Werner,et al.  A comparison of the efficiency and accuracy of the quadratic configuration interaction (QCISD), coupled cluster (CCSD), and Brueckner coupled cluster (BCCD) methods , 1992 .

[59]  Edward F. Valeev,et al.  Variational formulation of perturbative explicitly-correlated coupled-cluster methods. , 2008, Physical chemistry chemical physics : PCCP.

[60]  Frank Neese,et al.  Sparse maps—A systematic infrastructure for reduced-scaling electronic structure methods. I. An efficient and simple linear scaling local MP2 method that uses an intermediate basis of pair natural orbitals. , 2015, The Journal of chemical physics.

[61]  Tomasz Janowski,et al.  Array files for computational chemistry: MP2 energies , 2007, J. Comput. Chem..

[62]  R J Bartlett,et al.  Parallel implementation of electronic structure energy, gradient, and Hessian calculations. , 2008, The Journal of chemical physics.

[63]  D. Tew,et al.  Explicitly correlated PNO-MP2 and PNO-CCSD and their application to the S66 set and large molecular systems. , 2014, Physical chemistry chemical physics : PCCP.

[64]  Jürgen Gauss,et al.  Parallel Calculation of CCSD and CCSD(T) Analytic First and Second Derivatives. , 2008, Journal of chemical theory and computation.

[65]  A Eugene DePrince,et al.  Accuracy and Efficiency of Coupled-Cluster Theory Using Density Fitting/Cholesky Decomposition, Frozen Natural Orbitals, and a t1-Transformed Hamiltonian. , 2013, Journal of chemical theory and computation.

[66]  Frank Neese,et al.  SparseMaps--A systematic infrastructure for reduced-scaling electronic structure methods. IV. Linear-scaling second-order explicitly correlated energy with pair natural orbitals. , 2016, The Journal of chemical physics.

[67]  Mark S. Gordon,et al.  A Novel Approach to Parallel Coupled Cluster Calculations: Combining Distributed and Shared Memory Techniques for Modern Cluster Based Systems , 2007 .

[68]  Frank Neese,et al.  Sparse maps--A systematic infrastructure for reduced-scaling electronic structure methods. II. Linear scaling domain based pair natural orbital coupled cluster theory. , 2016, The Journal of chemical physics.

[69]  Trygve Helgaker,et al.  A direct atomic orbital driven implementation of the coupled cluster singles and doubles (CCSD) model , 1994 .

[70]  Karol Kowalski,et al.  Efficient computer implementation of the renormalized coupled-cluster methods: The R-CCSD[T], R-CCSD(T), CR-CCSD[T], and CR-CCSD(T) approaches , 2002 .

[71]  Kirk A Peterson,et al.  Optimized auxiliary basis sets for explicitly correlated methods. , 2008, The Journal of chemical physics.

[72]  M. Swart,et al.  π-π stacking tackled with density functional theory , 2007, Journal of molecular modeling.

[73]  Manoj K. Kesharwani,et al.  Some Observations on Counterpoise Corrections for Explicitly Correlated Calculations on Noncovalent Interactions. , 2014, Journal of chemical theory and computation.

[74]  Peter Pulay,et al.  An efficient reformulation of the closed‐shell self‐consistent electron pair theory , 1984 .

[75]  P Hobza,et al.  Noncovalent interactions: a challenge for experiment and theory. , 2000, Chemical reviews.

[76]  R. Bartlett,et al.  A direct product decomposition approach for symmetry exploitation in many-body methods. I. Energy calculations , 1991 .

[77]  Michael Dolg,et al.  Fully automated implementation of the incremental scheme: application to CCSD energies for hydrocarbons and transition metal compounds. , 2007, The Journal of chemical physics.

[78]  Henry F. Schaefer,et al.  Accelerating the convergence of the coupled-cluster approach: The use of the DIIS method , 1986 .

[79]  Justus A. Calvin,et al.  Clustered Low-Rank Tensor Format: Introduction and Application to Fast Construction of Hartree-Fock Exchange. , 2015, Journal of chemical theory and computation.

[80]  Hans-Joachim Werner,et al.  Low-order scaling local electron correlation methods. IV. Linear scaling local coupled-cluster (LCCSD) , 2001 .

[81]  Frederick R Manby,et al.  General orbital invariant MP2-F12 theory. , 2007, The Journal of chemical physics.

[82]  Peter Pulay,et al.  Localizability of dynamic electron correlation , 1983 .

[83]  Hans-Joachim Werner,et al.  Local treatment of electron correlation in coupled cluster theory , 1996 .

[84]  Robert J. Harrison,et al.  MADNESS: A Multiresolution, Adaptive Numerical Environment for Scientific Simulation , 2015, SIAM J. Sci. Comput..

[85]  Marcin Ziółkowski,et al.  A Locality Analysis of the Divide-Expand-Consolidate Coupled Cluster Amplitude Equations. , 2011, Journal of chemical theory and computation.

[86]  K. Kowalski,et al.  Toward enabling large-scale open-shell equation-of-motion coupled cluster calculations: triplet states of β-carotene. , 2014, The journal of physical chemistry. A.

[87]  Edward F. Valeev.  Improving on the resolution of the identity in linear R12 ab initio theories , 2004 .

[88]  John F. Stanton,et al.  A massively parallel tensor contraction framework for coupled-cluster computations , 2014, J. Parallel Distributed Comput..

[89]  Yang,et al.  Direct calculation of electron density in density-functional theory. , 1991, Physical review letters.