Enhancing Scalability and Load Balancing of Parallel Selected Inversion via Tree-Based Asynchronous Communication

We develop a method for improving the parallel scalability of computations that involve asynchronous task execution. We apply this method to the recently developed parallel selected inversion algorithm [Jacquelin, Lin and Yang 2014], named PSelInv, on massively parallel distributed memory machines. In the PSelInv method, we compute selected elements of the inverse of a sparse matrix A that can be decomposed as A = LU, where L is lower triangular and U is upper triangular. Computing these selected elements of A-1 requires restricted collective communications among a subset of processors within each column or row communication group created by a block cyclic distribution of L and U. We describe how this type of restricted collective communication can be implemented using asynchronous point-to-point MPI communications combined with a binary tree based data propagation scheme. Because multiple restricted collective communications may take place at the same time, we need to use a heuristic to prevent processors participating in multiple collective communications from receiving too many messages. This heuristic allows us to reduce communication load imbalance and improve the overall scalability of the selected inversion algorithm. For instance, when 6, 400 processors are used, we observe that the use of this heuristic leads to over 5x speedup for a number of test matrices. It also mitigates the performance variability introduced by an inhomogeneous network topology.

[1]  Mathias Jacquelin,et al.  PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion , 2017, ACM Trans. Math. Softw..

[2]  Joseph W. H. Liu.  The role of elimination trees in sparse factorization , 1990 .

[3]  TIMOTHY A. DAVISyTechnical,et al.  Computing the Sparse Inverse Subset: an Inverse Multifrontal Approach , 1995 .

[4]  พงศ์ศักดิ์ บินสมประสงค์,et al.  FORMATION OF A SPARSE BUS IMPEDANCE MATRIX AND ITS APPLICATION TO SHORT CIRCUIT STUDY , 1980 .

[5]  Jean-Yves L'Excellent,et al.  On Computing Inverse Entries of a Sparse Matrix in an Out-of-Core Environment , 2012, SIAM J. Sci. Comput..

[6]  Lexing Ying,et al.  SelInv---An Algorithm for Selected Inversion of a Sparse Symmetric Matrix , 2011, TOMS.

[7]  Eric Darve,et al.  A hybrid method for the parallel computation of Green's functions , 2009, J. Comput. Phys..

[8]  M. P. Anantram,et al.  A nested dissection approach to modeling transport in nanodevices: Algorithms and applications , 2013, 1305.1070.

[9]  James Demmel,et al.  SuperLU_DIST: A scalable distributed-memory sparse direct solver for unsymmetric linear systems , 2003, TOMS.

[10]  E. Weinan,et al.  Fast algorithm for extracting the diagonal of the inverse matrix with application to the electronic structure analysis of metallic systems , 2009 .

[11]  Eric Darve,et al.  A fast algorithm for sparse matrix computations related to inversion , 2013, J. Comput. Phys..

[12]  Eric Darve,et al.  Computing entries of the inverse of a sparse matrix using the FIND algorithm , 2008, J. Comput. Phys..

[13]  François-Henry Rouet,et al.  Parallel Computation of Entries of A-1 , 2015, SIAM J. Sci. Comput..

[14]  W. F. Tinney,et al.  On computing certain elements of the inverse of a sparse matrix , 1975, Commun. ACM.

[15]  Chao Yang,et al.  Accelerating atomic orbital-based electronic structure calculation via pole expansion and selected inversion , 2012, Journal of physics. Condensed matter : an Institute of Physics journal.

[16]  W. Kohn,et al.  Self-Consistent Equations Including Exchange and Correlation Effects , 1965 .

[17]  E Weinan,et al.  A Fast Parallel Algorithm for Selected Inversion of Structured Sparse Matrices with Application to 2D Electronic Structure Calculations , 2010, SIAM J. Sci. Comput..

[18]  E Weinan,et al.  Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation , 2011, J. Comput. Phys..

[19]  E Weinan,et al.  Pole-Based approximation of the Fermi-Dirac function , 2009, 0906.1319.

[20]  Eric Darve,et al.  Extension and optimization of the FIND algorithm: Computing Green's and less-than Green's functions , 2011, J. Comput. Phys..

[21]  Timothy A. Davis,et al.  The university of Florida sparse matrix collection , 2011, TOMS.