Up to 700k GPU Cores, Kepler, and the Exascale Future for Simulations of Star Clusters Around Black Holes

We present benchmarks on high precision direct astrophysical N-body simulations using up to several 100k GPU cores; their soft and strong scaling behaves very well at that scale and allows further increase of the core number in the future path to Exascale computing. Our simulations use large GPU clusters both in China (Chinese Academy of Sciences) as well as in Germany (Judge/Milkyway cluster at FZ Julich). Also we present first results on the performance gain by the new Kepler K20 GPU technology, which we have tested in two small experimental systems, and which also runs in the titan supercomputer in the United States, currently the fastest computer in the world. Our high resolution astrophysical N-body simulations are used for simulations of star clusters and galactic nuclei with central black holes. Some key issues in theoretical physics and astrophysics are addressed with them, such as galaxy formation and evolution, massive black hole formation, gravitational wave emission. The models have to cover thousands or more orbital time scales for the order of several million bodies. The total numerical effort is comparable if not higher than for the more widely known cosmological N-body simulations. Due to a complex structure in time (hierarchical blocked time steps) our codes are not considered “brute force”.

[1]  Junichiro Makino,et al.  Performance analysis of direct N-body calculations , 1988 .

[2]  Keigo Nitadori,et al.  Accelerating nbody6 with graphics processing units , 2012, 1205.1222.

[3]  David Merritt,et al.  Systolic and Hyper-Systolic Algorithms for the Gravitational N-Body Problem, with an Application to Brownian Motion , 2001, ArXiv.

[4]  Yunfei Chen,et al.  GPU accelerated molecular dynamics simulation of thermal conductivities , 2007, J. Comput. Phys..

[5]  Oliver Porth,et al.  Evolution of growing black holes in axisymmetric galaxy cores , 2011, 1108.3993.

[6]  Andreas Just,et al.  MERGERS OF UNEQUAL-MASS GALAXIES: SUPERMASSIVE BLACK HOLE BINARY EVOLUTION AND STRUCTURE OF MERGER REMNANTS , 2012, 1202.2124.

[7]  Peter Berczik,et al.  Brownian Motion of Black Holes in Dense Nuclei , 2004, astro-ph/0408029.

[8]  R. Spurzem.  Direct N-body simulations , 1999, astro-ph/9906154.

[9]  Zoltán Fodor,et al.  Lattice QCD as a video game , 2007, Comput. Phys. Commun..

[10]  Sverre J. Aarseth,et al.  Star Cluster Simulations: the State of the Art , 1999, astro-ph/9901069.

[11]  D. Merritt,et al.  Performance Analysis of Direct N-Body Algorithms on Special-Purpose Supercomputers , 2006, astro-ph/0608125.

[12]  Leon Cohen,et al.  A numerical integration scheme for the N-body gravitational problem , 1973 .

[13]  Hans-Peter Bischof,et al.  EFFICIENT MERGER OF BINARY SUPERMASSIVE BLACK HOLES IN NON- AXISYMMETRIC GALAXIES , 2006 .

[14]  Andreas Just,et al.  ENHANCED ACCRETION RATES OF STARS ON SUPERMASSIVE BLACK HOLES BY STAR–DISK INTERACTIONS IN GALACTIC NUCLEI , 2012, 1208.4954.

[15]  Junichiro Makino,et al.  On a Hermite Integrator with Ahmad-Cohen Scheme for Gravitational Many-Body Problems , 1992 .

[16]  P. Berczik,et al.  Calibration of radii and masses of open clusters with a simulation , 2010, 1009.0710.

[17]  S. Aarseth.  From NBODY1 to NBODY6: The Growth of an Industry , 1999 .

[18]  P. Berczik,et al.  Quantitative analysis of clumps in the tidal tails of star clusters , 2008, 0808.3293.

[19]  J. Makino,et al.  Sixth- and eighth-order Hermite integrator for N-body simulations , 2007, 0708.0738.

[20]  N. V. Kharchenko,et al.  Shape parameters of Galactic open clusters , 2008, 0812.3542.

[21]  Junichiro Makino,et al.  A Modified Aarseth Code for GRAPE and Vector Processors , 1991 .

[22]  Alessia Gualandris,et al.  Ejection of Supermassive Black Holes from Galaxy Cores , 2007, 0708.0771.

[23]  Sverre J. Aarseth,et al.  Gravitational N-Body Simulations , 2003 .

[24]  P. Berczik,et al.  Post-Newtonian simulations of super-massive black hole binaries in galactic nuclei , 2008, 0810.5253.

[25]  Junichiro Makino,et al.  Optimal order and time-step criterion for Aarseth-type N-body integrators , 1991 .

[26]  Andreas Just,et al.  FORMATION AND HARDENING OF SUPERMASSIVE BLACK HOLE BINARIES IN MINOR MERGERS OF DISK GALAXIES , 2012, 1203.1623.

[27]  Leslie Greengard,et al.  A fast algorithm for particle simulations , 1987 .

[28]  Rainer Spurzem,et al.  Long-Term Evolution of Massive Black Hole Binaries. II. Binary Evolution in Low-Density Galaxies , 2005, astro-ph/0507260.

[29]  Tsuyoshi Hamada,et al.  The Chamomile Scheme: An Optimized Algorithm for N-body simulations on Programmable Graphics Processing Units , 2007 .

[30]  Rainer Spurzem,et al.  Merger of massive black holes using N-body simulations with post-Newtonian corrections , 2008, 0811.3501.

[31]  Wei Ge,et al.  High performance massively parallel direct N-body simulations on large GPU clusters , 2011 .

[32]  Rainer Spurzem,et al.  INTERACTION OF RECOILING SUPERMASSIVE BLACK HOLES WITH STARS IN GALACTIC NUCLEI , 2012, 1201.3407.

[33]  P. Berczik,et al.  Dynamical friction of massive objects in galactic centres , 2010, 1009.2455.

[34]  Rainer Spurzem,et al.  BINARY BLACK HOLE MERGER IN GALACTIC NUCLEI: POST-NEWTONIAN SIMULATIONS , 2008, 0812.2756.

[35]  K. Holley-Bockelmann,et al.  SUPERMASSIVE BLACK HOLE BINARY EVOLUTION IN AXISYMMETRIC GALAXIES: THE FINAL PARSEC PROBLEM IS NOT A PROBLEM , 2013, 1302.1871.

[36]  P. Berczik,et al.  Simulations of the Hyades , 2011, 1110.1274.

[37]  Koji Yasuda,et al.  Two‐electron integral evaluation on the graphics processor unit , 2008, J. Comput. Chem..

[38]  Rainer Spurzem,et al.  FAST COALESCENCE OF MASSIVE BLACK HOLE BINARIES FROM MERGERS OF GALACTIC NUCLEI: IMPLICATIONS FOR LOW-FREQUENCY GRAVITATIONAL-WAVE ASTROPHYSICS , 2011, 1102.4855.