Developing a Multi-GPU-Enabled Preconditioned GMRES with Inexact Triangular Solves for Block Sparse Matrices

Solving triangular systems is the building block for preconditioned GMRES algorithm. Inexact preconditioning becomes attractive because of the feature of high parallelism on accelerators. In this paper, we propose and implement an iterative, inexact block triangular solve on multi-GPUs based on PETSc’s framework. In addition, by developing a distributed block sparse matrix-vector multiplication procedure and investigating the optimized vector operations, we form the multi-GPU-enabled preconditioned GMRES with the block Jacobi preconditioner. In the implementation, the GPU-Direct technique is employed to avoid host-device memory copies. The preconditioning step used by PETSc’s structure and the cuSPARSE library are also investigated for performance comparisons. The experiments show that the developed GMRES with inexact preconditioning on 8 GPUs can achieve up to 4.4x speedup over the CPU-only implementation with exact preconditioning using 8 MPI processes.

[1]  Jonas Koko,et al.  Parallel preconditioned conjugate gradient algorithm on GPU , 2012, J. Comput. Appl. Math..

[2]  Edmond Chow,et al.  Fine-Grained Parallel Incomplete LU Factorization , 2015, SIAM J. Sci. Comput..

[3]  Wolfgang Straßer,et al.  A Parallel Preconditioned Conjugate Gradient Solver for the Poisson Problem on a Multi-GPU Platform , 2010, 2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing.

[4]  Matthew G. Knepley,et al.  Preliminary Implementation of PETSc Using GPUs , 2013 .

[5]  Yousef Saad,et al.  GPU-accelerated preconditioned iterative linear solvers , 2013, The Journal of Supercomputing.

[6]  Y. Saad,et al.  Overlapping Domain Decomposition Algorithms for General Sparse Matrices , 1996, Numer. Linear Algebra Appl..

[7]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[8]  Karl Rupp,et al.  ViennaCL - Linear Algebra Library for Multi- and Many-Core Architectures , 2016, SIAM J. Sci. Comput..

[9]  Jacques M. Bahi,et al.  Parallel sparse linear solver with GMRES method using minimization techniques of communications for GPU clusters , 2014, The Journal of Supercomputing.

[10]  Enrique S. Quintana-Ortí,et al.  An efficient GPU version of the preconditioned GMRES method , 2018, The Journal of Supercomputing.

[11]  Yaohang Li,et al.  An Implementation of Block Conjugate Gradient Algorithm on CPU-GPU Processors , 2014, 2014 Hardware-Software Co-Design for High Performance Computing.

[12]  Mark Hoemmen,et al.  Optimization of Block Sparse Matrix-Vector Multiplication on Shared-Memory Parallel Architectures , 2016, 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW).

[13]  Sheldon X.-D. Tan,et al.  Parallel GMRES solver for fast analysis of large linear dynamic systems on GPU platforms , 2016, Integr..

[14]  Jack J. Dongarra,et al.  Optimizing Krylov Subspace Solvers on Graphics Processing Units , 2014, 2014 IEEE International Parallel & Distributed Processing Symposium Workshops.

[15]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[16]  Sheldon X.-D. Tan,et al.  Parallel power grid analysis using preconditioned GMRES solver on CPU-GPU platforms , 2013, 2013 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[17]  Timothy A. Davis,et al.  The university of Florida sparse matrix collection , 2011, TOMS.

[18]  Jack J. Dongarra,et al.  Improving the Performance of CA-GMRES on Multicores with Multiple GPUs , 2014, 2014 IEEE 28th International Parallel and Distributed Processing Symposium.

[19]  Viviane Cristine Silva,et al.  Performance Analysis of Multi-GPU Implementations of Krylov-Subspace Methods Applied to FEA of Electromagnetic Phenomena , 2015, IEEE Transactions on Magnetics.

[20]  F. Marcuzzi,et al.  Fully iterative ILU preconditioning of the unsteady Navier-Stokes equations for GPGPU , 2019, Comput. Math. Appl..

[21]  Edmond Chow,et al.  Iterative Sparse Triangular Solves for Preconditioning , 2015, Euro-Par.

[22]  Santa Clara,et al.  Parallel Solution of Sparse Triangular Linear Systems in the Preconditioned Iterative Methods on the GPU , 2011 .

[23]  Zhangxin Chen,et al.  Accelerating the GMRES Solver with Block ILU (K) Preconditioner on GPUs in Reservoir Simulation , 2014 .

[24]  Yushun Wang,et al.  GPU-accelerated preconditioned GMRES method for two-dimensional Maxwell's equations , 2017, Int. J. Comput. Math..

[25]  Jiaquan Gao,et al.  A multi-GPU parallel optimization model for the preconditioned conjugate gradient algorithm , 2017, Parallel Comput..

[26]  Maxime R. Hugues,et al.  A Communication Optimization Scheme for Basis Computation of Krylov Subspace Methods on Multi-GPUs , 2014, VECPAR.

[27]  Edmond Chow,et al.  Domain Overlap for Iterative Sparse Triangular Solves on GPUs , 2016, Software for Exascale Computing.

[28]  Sivakumaran Nadarajah,et al.  Fine-grain Parallel Smoothing by Asynchronous Iterations and Incomplete Sparse Approximate Inverses for Computational Fluid Dynamics , 2020 .

[29]  Sivasankaran Rajamanickam,et al.  Domain Decomposition Preconditioners for Communication-Avoiding Krylov Methods on a Hybrid CPU/GPU Cluster , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[30]  Zhangxin Chen,et al.  GPU-Accelerated Preconditioned GMRES Solver , 2016, 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS).

[31]  Edmond Chow,et al.  Asynchronous Iterative Algorithm for Computing Incomplete Factorizations on GPUs , 2015, ISC.