Hybrid MPI/OpenMP Parallelization in FETI-DP Methods

We present an approach to hybrid MPI/OpenMP parallelization in FETI-DP methods using OpenMP with PETSc+MPI in the finite element assembly and using the shared memory parallel direct solver Pardiso in the FETI-DP solution phase. Our approach thus uses OpenMP parallelization on subdomains and MPI in between subdomains. We investigate the efficiency of this approach for a benchmark problem from two dimensional nonlinear hyperelasticity. We observe good scalability for up to four threads for each MPI rank on a state-of-the-art Ivy Bridge architecture and incremental improvements for up to ten OpenMP threads for each MPI rank.

[1]  Axel Klawonn,et al.  Towards Extremely Scalable Nonlinear Domain Decomposition Methods for Elliptic Partial Differential Equation , 2014 .

[2]  William Gropp,et al.  Efficient Management of Parallelism in Object-Oriented Numerical Software Libraries , 1997, SciTools.

[3]  Barry F. Smith,et al.  Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations , 1996 .

[4]  A. Klawonn,et al.  Highly scalable parallel domain decomposition methods with an application to biomechanics , 2010 .

[5]  Oliver Rheinbach,et al.  Parallel Iterative Substructuring in Structural Mechanics , 2009 .

[6]  Timothy A. Davis,et al.  A combined unifrontal/multifrontal method for unsymmetric sparse matrices , 1999, TOMS.

[7]  Gerhard Wellein,et al.  LIKWID: A Lightweight Performance-Oriented Tool Suite for x86 Multicore Environments , 2010, 2010 39th International Conference on Parallel Processing Workshops.

[8]  Olof B. Widlund,et al.  Dual‐primal FETI methods for linear elasticity , 2006 .

[9]  Gerhard Wellein,et al.  LIKWID: Lightweight Performance Tools , 2011, CHPC.

[10]  Patrick Amestoy,et al.  A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling , 2001, SIAM J. Matrix Anal. Appl..

[11]  Frédéric Feyel,et al.  A parallel algorithm for direct solution of large sparse linear systems, well suitable to domain decomposition methods , 2009 .

[12]  Andrea Toselli,et al.  Domain decomposition methods : algorithms and theory , 2005 .

[13]  Matthew G. Knepley,et al.  PETSc Users Manual: Revision 3.11 , 2019 .

[14]  Charbel Farhat,et al.  Salinas: A Scalable Software for High-Performance Structural and Solid Mechanics Simulations , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[15]  Mike Ashworth,et al.  Developing Hybrid OpenMP-MPI Parallelism for Fluidity-Next Generation Geophysical Fluid Modelling Technology , 2012 .

[16]  O. C. Zienkiewicz,et al.  The Finite Element Method for Solid and Structural Mechanics , 2013 .

[17]  C. Farhat,et al.  A scalable dual-primal domain decomposition method , 2000, Numer. Linear Algebra Appl..

[18]  Timothy A. Davis,et al.  A column pre-ordering strategy for the unsymmetric-pattern multifrontal method , 2004, TOMS.

[19]  Timothy A. Davis,et al.  An Unsymmetric-pattern Multifrontal Method for Sparse Lu Factorization , 1993 .

[20]  Rudolf A. Römer,et al.  On large‐scale diagonalization techniques for the Anderson model of localization , 2005, SIAM J. Sci. Comput..

[21]  Ibrahima Gueye,et al.  A new parallel sparse direct solver: Presentation and numerical experiments in large‐scale structural mechanics parallel computing , 2011 .

[22]  D. Rixen,et al.  FETI‐DP: a dual–primal unified FETI method—part I: A faster alternative to the two‐level FETI method , 2001 .

[23]  Ibrahima Gueye,et al.  Résolution de grands systèmes linéaires issus de la méthode des éléments finis sur des calculateurs massivement parallèles. (Solving large linear systems arising in finite element approximations on massively parallel computers) , 2009 .

[24]  Olaf Schenk,et al.  Matching-based preprocessing algorithms to the solution of saddle-point problems in large-scale nonconvex interior-point optimization , 2007, Comput. Optim. Appl..

[25]  Axel Klawonn,et al.  A Nonlinear FETI-DP Method with an Inexact Coarse Problem , 2016 .

[26]  Patrick R. Amestoy,et al.  Multifrontal parallel distributed symmetric and unsymmetric solvers , 2000 .

[27]  Axel Klawonn,et al.  Inexact FETI‐DP methods , 2007 .

[28]  Olaf Schenk,et al.  Fast Methods for Computing Selected Elements of the Green's Function in Massively Parallel Nanoelectronic Device Simulations , 2013, Euro-Par.