CHOLLA: A NEW MASSIVELY PARALLEL HYDRODYNAMICS CODE FOR ASTROPHYSICAL SIMULATION

We present Cholla (Computational Hydrodynamics On ParaLLel Architectures), a new three-dimensional hydrodynamics code that harnesses the power of graphics processing units (GPUs) to accelerate astrophysical simulations. Cholla models the Euler equations on a static mesh using state-of-the-art techniques, including the unsplit Corner Transport Upwind (CTU) algorithm, a variety of exact and approximate Riemann solvers, and multiple spatial reconstruction techniques including the piecewise parabolic method (PPM). Using GPUs, Cholla evolves the fluid properties of thousands of cells simultaneously and can update over ten million cells per GPU-second while using an exact Riemann solver and PPM reconstruction. Owing to the massively-parallel architecture of GPUs and the design of the Cholla code, astrophysical simulations with physically interesting grid resolutions (> 256^3) can easily be computed on a single device. We use the Message Passing Interface library to extend calculations onto multiple devices and demonstrate nearly ideal scaling beyond 64 GPUs. A suite of test problems highlights the physical accuracy of our modeling and provides a useful comparison to other codes. We then use Cholla to simulate the interaction of a shock wave with a gas cloud in the interstellar medium, showing that the evolution of the cloud is highly dependent on its density structure. We reconcile the computed mixing time of a turbulent cloud with a realistic density distribution destroyed by a strong shock with the existing analytic theory for spherical cloud destruction by describing the system in terms of its median gas density.

[1]  Richard Sanders,et al.  Multidimensional Dissipation for Upwind Schemes , 1998 .

[2]  B. Fryxell,et al.  FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes , 2000 .

[3]  R. LeVeque Finite Volume Methods for Hyperbolic Problems: Characteristics and Riemann Problems for Linear Hyperbolic Equations , 2002 .

[4]  J. Stone,et al.  The hydrodynamics of shock-cloud interactions in three dimensions , 1995 .

[5]  Peng Wang,et al.  Adaptive mesh fluid simulations on GPU , 2009, 0910.5547.

[6]  R. Klein,et al.  On the hydrodynamic interaction of shock waves with interstellar clouds. 1: Nonradiative shocks in small clouds , 1994 .

[7]  Igor Kulikov,et al.  GPUPEGAS: A NEW GPU-ACCELERATED HYDRODYNAMIC CODE FOR NUMERICAL SIMULATIONS OF INTERACTING GALAXIES , 2014 .

[8]  P. Huynh,et al.  HERACLES: a three-dimensional radiation hydrodynamics code , 2007 .

[9]  Properties of Galactic Outflows: Measurements of the Feedback from Star Formation , 1998, astro-ph/9810233.

[10]  P. Woodward,et al.  The Piecewise Parabolic Method (PPM) for Gas Dynamical Simulations , 1984 .

[11]  Wai How Hui,et al.  A unified coordinate system for solving the three-dimensional Euler equations , 1999 .

[12]  R. Teyssier Cosmological hydrodynamics with adaptive mesh refinement - A new high resolution code called RAMSES , 2001, astro-ph/0111367.

[13]  Nickolay Y. Gnedin,et al.  Computational Eulerian hydrodynamics and Galilean invariance , 2009, 0909.0513.

[14]  Toshikazu Ebisuzaki,et al.  A special-purpose computer for gravitational many-body problems , 1990, Nature.

[15]  Richard Liska,et al.  Comparison of Several Difference Schemes on 1D and 2D Test Problems for the Euler Equations , 2003, SIAM J. Sci. Comput..

[16]  James J. Quirk,et al.  A Contribution to the Great Riemann Solver Debate , 1994 .

[17]  Crushing of interstellar gas clouds in supernova remnants. I. The role of thermal conduction and radiative losses , 2005, astro-ph/0508638.

[18]  P. Anninos,et al.  Magnetohydrodynamic Simulations of Shock Interactions with Radiative Clouds , 2004, astro-ph/0410285.

[19]  D. Merritt,et al.  Performance Analysis of Direct N-Body Algorithms on Special-Purpose Supercomputers , 2006, astro-ph/0608125.

[20]  R. Klein,et al.  On the Hydrodynamic Interaction of Shock Waves with Interstellar Clouds. II. The Effect of Smooth Cloud Boundaries on Cloud Destruction and Cloud Turbulence , 2005, astro-ph/0511016.

[21]  R. Spurzem Direct N-body simulations , 1999, astro-ph/9906154.

[22]  A. Kravtsov,et al.  TOWARD A COMPLETE ACCOUNTING OF ENERGY AND MOMENTUM FROM STELLAR FEEDBACK IN GALAXY FORMATION SIMULATIONS , 2012, 1210.4957.

[23]  Devin W. Silvia,et al.  ENZO: AN ADAPTIVE MESH REFINEMENT CODE FOR ASTROPHYSICS , 2013, J. Open Source Softw..

[24]  A. N. V. K. Ravtsov,et al.  TOWARDS A COMPLETE ACCOUNTING OF ENERGY AND MOMENTUM FROM STELLAR FEEDBACK IN GALAXY FORMATION SIMULATIONS , 2012 .

[25]  G. Sod A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws , 1978 .

[26]  Phillip Colella,et al.  A limiter for PPM that preserves accuracy at smooth extrema , 2008, J. Comput. Phys..

[27]  Corporate The MPI Forum,et al.  MPI: a message passing interface , 1993, Supercomputing '93.

[28]  Phillip Colella,et al.  Efficient Solution Algorithms for the Riemann Problem for Real Gases , 1985 .

[29]  G. Strang On the Construction and Comparison of Difference Schemes , 1968 .

[30]  James Binney,et al.  Multi‐level adaptive particle mesh (MLAPM): a c code for cosmological simulations , 2001, astro-ph/0103503.

[31]  P. Teuben,et al.  Athena: A New Code for Astrophysical MHD , 2008, 0804.0402.

[32]  S. Chandrasekhar Hydrodynamic and Hydromagnetic Stability , 1961 .

[33]  P. Colella Multidimensional upwind methods for hyperbolic conservation laws , 1990 .

[34]  I. Bohachevsky,et al.  Finite difference method for numerical computation of discontinuous solutions of the equations of fluid dynamics , 1959 .

[35]  Moncho Gómez-Gesteira,et al.  New multi-GPU implementation for smoothed particle hydrodynamics on heterogeneous clusters , 2013, Comput. Phys. Commun..

[36]  Bernd Einfeld On Godunov-type methods for gas dynamics , 1988 .

[37]  P. Colella,et al.  Local adaptive mesh refinement for shock hydrodynamics , 1989 .

[38]  J. Saltzman,et al.  An unsplit 3D upwind method for hyperbolic conservation laws , 1994 .

[39]  M. L. Norman,et al.  Simulating Radiating and Magnetized Flows in Multiple Dimensions with ZEUS-MP , 2005, astro-ph/0511545.

[40]  A. Raga,et al.  Star formation triggered by SN explosions: an application to the stellar association of β Pictoris , 2006, 0711.0105.

[41]  Junichiro Makino,et al.  Formation of massive black holes through runaway collisions in dense young star clusters , 2004, Nature.

[42]  B. V. Leer,et al.  Towards the ultimate conservative difference scheme. IV. A new approach to numerical convection , 1977 .

[43]  Jack Dongarra,et al.  Special Issue on Program Generation, Optimization, and Platform Adaptation , 2005, Proc. IEEE.

[44]  P. Padoan,et al.  The Stellar Initial Mass Function from Turbulent Fragmentation , 2000, astro-ph/0011465.

[45]  Richard Sanders,et al.  Regular ArticleMultidimensional Dissipation for Upwind Schemes: Stability and Applications to Gas Dynamics☆ , 1998 .

[46]  W. F. Noh Errors for calculations of strong shocks using an artificial viscosity and artificial heat flux , 1985 .

[47]  R. Sutherland,et al.  STARBURST-DRIVEN GALACTIC WINDS: FILAMENT FORMATION AND EMISSION PROCESSES , 2009, 0907.4004.

[48]  S. Aarseth From NBODY1 to NBODY6: The Growth of an Industry , 1999 .

[49]  Dinshaw Balsara,et al.  Second-Order-accurate Schemes for Magnetohydrodynamics with Divergence-free Reconstruction , 2003, astro-ph/0308249.

[50]  R. Klessen,et al.  Control of star formation by supersonic turbulence , 2000, astro-ph/0301093.

[51]  B. V. Leer,et al.  Towards the ultimate conservative difference scheme V. A second-order sequel to Godunov's method , 1979 .

[52]  L. Cowie,et al.  The interaction between the blast wave of a supernova remnant and interstellar clouds. , 1975 .

[53]  M. Norman,et al.  Shock interactions with magnetized interstellar clouds. 1: Steady shocks hitting nonradiative clouds , 1994 .

[54]  E. Toro Riemann Solvers and Numerical Methods for Fluid Dynamics , 1997 .

[55]  Message P Forum,et al.  MPI: A Message-Passing Interface Standard , 1994 .

[56]  B. Robertson,et al.  ADIABATIC HEATING OF CONTRACTING TURBULENT FLUIDS , 2012, 1203.4815.

[57]  G. Snyder,et al.  The Magnetohydrodynamics of Shock-Cloud Interaction in Three Dimensions , 2008, 0802.2708.

[58]  S. Osher,et al.  Efficient implementation of essentially non-oscillatory shock-capturing schemes,II , 1989 .

[59]  James M. Stone,et al.  An unsplit Godunov method for ideal MHD via constrained transport in three dimensions , 2007, J. Comput. Phys..

[60]  E. Laitone,et al.  Elements of Gasdynamics. , 1957 .

[61]  P. Roe,et al.  On Godunov-type methods near low densities , 1991 .

[62]  Tzihong Chiueh,et al.  GAMER: A GRAPHIC PROCESSING UNIT ACCELERATED ADAPTIVE-MESH-REFINEMENT CODE FOR ASTROPHYSICS , 2009, 0907.3390.