Development of a discontinuous Galerkin solver using Legion for heterogeneous high-performance computing architectures

This work discusses the development, verification and performance assessment of a discontinuous Galerkin solver for the compressible Navier-Stokes equations using the Legion programming system. This is motivated by (i) the potential of this family of high-order numerical methods to accurately and efficiently realize scale-resolving simulations on unstructured grids and (ii) the desire to accommodate the utilization of emerging compute platforms that exhibit increased parallelism and heterogeneity. As a task-based programmingmodel specifically designed for performance portability across distributed heterogeneous architectures, Legion represents an interesting alternative to the traditional approach of using Message Passing Interface for massively parallel computational physics solvers. Following a detailed discussion of the implementation, the high-order convergence of the solver is demonstrated by a suite of canonical test cases and good strong scaling behavior is obtained. This work constitutes a first step towards a research platform that is able to be deployed and efficiently run on modern supercomputers.

[1]  E. Toro,et al.  Restoration of the contact surface in the HLL-Riemann solver , 1994 .

[2]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[3]  Ralf Hartmann,et al.  An optimal order interior penalty discontinuous Galerkin discretization of the compressible Navier-Stokes equations , 2008, J. Comput. Phys..

[4]  Chi-Wang Shu,et al.  High Order Strong Stability Preserving Time Discretizations , 2009, J. Sci. Comput..

[5]  Alexander Aiken,et al.  Legion: Expressing locality and independence with logical regions , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[6]  Rémi Abgrall,et al.  High‐order CFD methods: current status and perspective , 2013 .

[7]  J. Debonis.  Solutions of the Taylor-Green Vortex Problem Using High-Resolution Explicit Finite Difference Methods , 2013 .

[8]  Daniel Sunderland,et al.  Kokkos: Enabling manycore performance portability through polymorphic memory access patterns , 2014, J. Parallel Distributed Comput..

[9]  William Gropp,et al.  CFD Vision 2030 Study: A Path to Revolutionary Computational Aerosciences , 2014 .

[10]  Freddie D. Witherden,et al.  PyFR: An open source framework for solving advection-diffusion type problems on streaming architectures using the flux reconstruction approach , 2013, Comput. Phys. Commun..

[11]  L. Bricteux,et al.  Implicit LES of free and wall‐bounded turbulent flows based on the discontinuous Galerkin/symmetric interior penalty method , 2015 .

[12]  C. Munz,et al.  A space–time adaptive discontinuous Galerkin scheme , 2015 .

[13]  James R. DeBonis,et al.  A Survey of the Isentropic Euler Vortex Problem Using High-Order Methods , 2015 .

[14]  Alessandro Colombo,et al.  On the development of an implicit high-order Discontinuous Galerkin method for DNS and implicit LES of turbulent flows , 2016 .

[15]  Alexander Aiken,et al.  Dependent partitioning , 2016, OOPSLA.

[16]  V. Venkatakrishnan,et al.  On the role and challenges of CFD in the aerospace industry , 2016, The Aeronautical Journal.

[17]  Freddie D. Witherden,et al.  High-Order Implicit Large-Eddy Simulations of Flow over a NACA0021 Aerofoil , 2017 .

[18]  Freddie D. Witherden,et al.  On the utility of GPU accelerated high-order methods for unsteady flow simulations: A comparison with industry-standard tools , 2017, J. Comput. Phys..

[19]  Michael Bauer,et al.  S3D-Legion : An Exascale Software for Direct Numerical Simulation of Turbulent Combustion with Complex Multicomponent Chemistry , 2017 .

[20]  Freddie D. Witherden,et al.  Future Directions in Computational Fluid Dynamics , 2017 .

[21]  Siva Nadarajah,et al.  A comprehensive high-order solver verification methodology for free fluid flows , 2017 .

[22]  Julien Loiseau,et al.  FleCSPH: a Parallel and Distributed Smoothed Particle Hydrodynamics Framework Based on FleCSI , 2018, 2018 International Conference on High Performance Computing & Simulation (HPCS).

[23]  Gianluca Iaccarino,et al.  Soleil-X : Turbulence, Particles, and Radiation in the Regent Programming Language , 2018 .

[24]  Matthias Ihme,et al.  On underresolved simulations of compressible turbulence using an entropy-bounded DG method: Solution stabilization, scheme optimization, and benchmark against a finite-volume solver , 2018 .

[25]  Aditya K. Pandare,et al.  A p -adaptive Discontinuous Galerkin Method for Compressible Flows Using Charm++ , 2020 .

[26]  Freddie D. Witherden,et al.  ZEFR: A GPU-accelerated high-order solver for compressible viscous flows using the flux reconstruction method , 2020, Comput. Phys. Commun..

[27]  Matthias K. Gobbert,et al.  A dynamically load-balanced parallel p-adaptive implicit high-order flux reconstruction method for under-resolved turbulence simulation , 2019, J. Comput. Phys..

[28]  Javier Urzay,et al.  HTR solver: An open-source exascale-oriented task-based multi-GPU high-order code for hypersonic aerothermodynamics , 2020, Comput. Phys. Commun..

[29]  J. Peraire,et al.  GPU-accelerated Large Eddy Simulation of Hypersonic Flows , 2020 .

[30]  Vincent Couaillier,et al.  A High-Order Multiscale Approach to Turbulence for Compact Nodal Schemes. , 2020, Computer methods in applied mechanics and engineering.

[31]  Gregor Gassner,et al.  FLEXI: A high order discontinuous Galerkin framework for hyperbolic-parabolic conservation laws , 2019, Comput. Math. Appl..