Optimizing the hybrid parallelization of BHAC

We present our experience with the modernization on the GR-MHD code BHAC, aimed at improving its novel hybrid (MPI+OpenMP) parallelization scheme. In doing so, we showcase the use of performance profiling tools usable on x86 (Intelbased) architectures. Our performance characterization and threading analysis provided guidance in improving the concurrency and thus the efficiency of the OpenMP parallel regions. We assess scaling and communication patterns in order to identify and alleviate MPI bottlenecks, with both runtime switches and precise code interventions. The performance of optimized version of BHAC improved by ∼ 28%, making it viable for scaling on hundreds of thousands of supercomputer nodes. We finally test whether porting such optimizations to different hardware is likewise beneficial on the new architecture by running on ARM A64FX vector nodes.

[1]  I. E. Mellah,et al.  MPI-AMRVAC 2.0 for Solar and Astrophysical Applications , 2017, 1710.06140.

[2]  P. Woodward,et al.  The Piecewise Parabolic Method (PPM) for Gas Dynamical Simulations , 1984 .

[3]  H. Rix,et al.  An astronomical institute’s perspective on meeting the challenges of the climate crisis , 2020, Nature Astronomy.

[4]  Bruno Costa,et al.  An improved WENO-Z scheme , 2016, J. Comput. Phys..

[5]  R. Keppens,et al.  MPI-AMRVAC FOR SOLAR AND ASTROPHYSICS , 2014, 1407.2052.

[6]  B. Ripperda,et al.  Reconnection and particle acceleration in interacting flux ropes - II. 3D effects on test particles in magnetically dominated plasmas , 2017, 1707.08920.

[7]  L. Rezzolla,et al.  Constrained transport and adaptive mesh refinement in the Black Hole Accretion Code , 2019, Astronomy & Astrophysics.

[8]  B. Ripperda,et al.  Magnetic Reconnection and Hot Spot Formation in Black Hole Accretion Disks , 2020, Astrophysical Journal.

[9]  B. A. Boom,et al.  GW170817: Observation of Gravitational Waves from a Binary Neutron Star Inspiral. , 2017, Physical review letters.

[10]  S. T. Timmer,et al.  First M87 Event Horizon Telescope Results. I. The Shadow of the Supermassive Black Hole , 2019, 1906.11238.

[11]  H. Falcke,et al.  How to tell an accreting boson star from a black hole , 2018, Monthly Notices of the Royal Astronomical Society.

[12]  R. Keppens,et al.  Relativistic resistive magnetohydrodynamic reconnection and plasmoid formation in merging flux tubes , 2018, Monthly Notices of the Royal Astronomical Society.

[13]  S. Rabien,et al.  Detection of orbital motions near the last stable circular orbit of the massive black hole SgrA* , 2018, Astronomy & Astrophysics.

[14]  Daniel C. M. Palumbo,et al.  First M87 Event Horizon Telescope Results. V. Physical Origin of the Asymmetric Ring , 2019, The Astrophysical Journal.

[15]  Simon Portegies Zwart,et al.  The ecological impact of high-performance computing in astrophysics , 2020, Nature Astronomy.

[16]  T. Paumard,et al.  EVIDENCE FOR X-RAY SYNCHROTRON EMISSION FROM SIMULTANEOUS MID-INFRARED TO X-RAY OBSERVATIONS OF A STRONG Sgr A* FLARE , 2009, 0903.3416.

[17]  Rony Keppens,et al.  Parallel, grid-adaptive approaches for relativistic hydro and magnetohydrodynamics , 2012, J. Comput. Phys..

[18]  Daniel C. M. Palumbo,et al.  The Event Horizon General Relativistic Magnetohydrodynamic Code Comparison Project , 2019, The Astrophysical Journal Supplement Series.

[19]  Gerhard Wellein,et al.  Performance Modeling of Streaming Kernels and Sparse Matrix-Vector Multiplication on A64FX , 2020, 2020 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS).

[20]  H. Falcke,et al.  The black hole accretion code , 2016, 1611.09720.

[21]  L. Rezzolla,et al.  General-relativistic Resistive Magnetohydrodynamics with Robust Primitive-variable Recovery for Accretion Disk Simulations , 2019, The Astrophysical Journal Supplement Series.

[22]  Herb Sutter,et al.  The Free Lunch Is Over A Fundamental Turn Toward Concurrency in Software , 2013 .