Performance of fully-coupled algebraic multigrid preconditioners for large-scale VMS resistive MHD

Abstract This work explores the current performance and scaling of a fully-implicit stabilized unstructured finite element (FE) variational multiscale (VMS) capability for large-scale simulations of 3D incompressible resistive magnetohydrodynamics (MHD). The large-scale linear systems that are generated by a Newton nonlinear solver approach are iteratively solved by preconditioned Krylov subspace methods. The efficiency of this approach is critically dependent on the scalability and performance of the algebraic multigrid preconditioner. This study considers the performance of the numerical methods as recently implemented in the second-generation Trilinos implementation that is 64-bit compliant and is not limited by the 32-bit global identifiers of the original Epetra-based Trilinos. The study presents representative results for a Poisson problem on 1.6 million cores of an IBM Blue Gene/Q platform to demonstrate very large-scale parallel execution. Additionally, results for a more challenging steady-state MHD generator and a transient solution of a benchmark MHD turbulence calculation for the full resistive MHD system are also presented. These results are obtained on up to 131,000 cores of a Cray XC40 and one million cores of a BG/Q system.

[1]  Paul Lin,et al.  Performance of fully coupled domain decomposition preconditioners for finite element transport/reaction simulations , 2005 .

[2]  Rajeev Thakur,et al.  Improving the Performance of Collective Operations in MPICH , 2003, PVM/MPI.

[3]  Jonathan J. Hu,et al.  Reducing Communication Costs for Sparse Matrix Multiplication within Algebraic Multigrid , 2015, SIAM J. Sci. Comput..

[4]  Paul Lin,et al.  A parallel fully coupled algebraic multilevel preconditioner applied to multiphysics PDE applications: Drift‐diffusion, flow/transport/reaction, resistive MHD , 2010 .

[5]  Yousef Saad,et al.  Hybrid Krylov Methods for Nonlinear Systems of Equations , 1990, SIAM J. Sci. Comput..

[6]  Paul T. Lin,et al.  Towards a scalable fully-implicit fully-coupled resistive MHD formulation with stabilized FE methods , 2009, J. Comput. Phys..

[7]  Paul Lin,et al.  Improving multigrid performance for unstructured mesh drift–diffusion simulations on 147,000 cores , 2012 .

[8]  Michael A. Heroux,et al.  Tpetra, and the use of generic programming in scientific computing , 2012 .

[9]  Luis Chacon,et al.  An optimal, parallel, fully implicit Newton–Krylov solver for three-dimensional viscoresistive magnetohydrodynamicsa) , 2008 .

[10]  D. Keyes,et al.  Jacobian-free Newton-Krylov methods: a survey of approaches and applications , 2004 .

[11]  Constantine Bekas,et al.  An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth's mantle , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[12]  Ray S. Tuminaro,et al.  A New Petrov--Galerkin Smoothed Aggregation Preconditioner for Nonsymmetric Linear Systems , 2008, SIAM J. Sci. Comput..

[13]  P. Mininni,et al.  Lack of universality in decaying magnetohydrodynamic turbulence. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[14]  T. Hughes,et al.  The variational multiscale method—a paradigm for computational mechanics , 1998 .

[15]  John N. Shadid,et al.  Scalable implicit incompressible resistive MHD with stabilized FE and fully-coupled Newton–Krylov-AMG , 2016 .

[16]  M. Benzi.  Preconditioning techniques for large linear systems: a survey , 2002 .

[17]  Paul T. Lin,et al.  Performance of a parallel algebraic multilevel preconditioner for stabilized finite element semiconductor device modeling , 2009, J. Comput. Phys..

[18]  T. Hughes.  Multiscale phenomena: Green's functions, the Dirichlet-to-Neumann formulation, subgrid scale models, bubbles and the origins of stabilized methods , 1995 .

[19]  Mehmet Deveci,et al.  Multi-Jagged: A Scalable Parallel Spatial Partitioning Algorithm , 2016, IEEE Transactions on Parallel and Distributed Systems.

[20]  William Gropp,et al.  Efficient Management of Parallelism in Object-Oriented Numerical Software Libraries , 1997, SciTools.

[21]  Paul Lin,et al.  Performance of fully coupled algebraic multilevel domain decomposition preconditioners for incompressible flow and transport , 2006 .

[22]  D. Rosenberg,et al.  The dynamics of unforced turbulence at high Reynolds number for Taylor–Green vortices generalized to MHD , 2009, 0906.1384.

[23]  David Sondak,et al.  A new class of finite element variational multiscale turbulence models for incompressible magnetohydrodynamics , 2014, J. Comput. Phys..

[24]  Mark Frederick Hoemmen,et al.  An Overview of Trilinos , 2003 .

[25]  Todd Gamblin,et al.  Scaling Algebraic Multigrid Solvers: On the Road to Exascale , 2010, CHPC.

[26]  Santiago Badia,et al.  Multilevel Balancing Domain Decomposition at Extreme Scales , 2016, SIAM J. Sci. Comput..

[27]  Daniel Sunderland,et al.  Manycore performance-portability: Kokkos multidimensional array library , 2012 .

[28]  Heidi K. Thornquist,et al.  Amesos2 and Belos: Direct and iterative solvers for large sparse linear systems , 2012 .

[29]  Sivasankaran Rajamanickam,et al.  Towards Extreme-Scale Simulations for Low Mach Fluids with Second-Generation Trilinos , 2014, Parallel Process. Lett..