Massively Parallel Stencil Strategies for Radiation Transport Moment Model Simulations

The radiation transport equation is a mesoscopic equation in high dimensional phase space. Moment methods approximate it via a system of partial differential equations in traditional space-time. One challenge is the high computational intensity due to large vector sizes (1 600 components for P39) in each spatial grid point. In this work, we extend the calculable domain size in 3D simulations considerably, by implementing the StaRMAP methodology within the massively parallel HPC framework NAStJA, which is designed to use current supercomputers efficiently. We apply several optimization techniques, including a new memory layout and explicit SIMD vectorization. We showcase a simulation with 200 billion degrees of freedom, and argue how the implementations can be extended and used in many scientific domains.

[1]  Kevin T. Clarno,et al.  Denovo: A New Three-Dimensional Parallel Discrete Ordinates Code in SCALE , 2010 .

[2]  Marco Sammartino,et al.  A thermodynamical approach to Eddington factors , 1991 .

[3]  James Paul Holloway,et al.  Two-dimensional time dependent Riemann solvers for neutron transport , 2005 .

[4]  Benjamin Seibold,et al.  Optimal prediction for moment models: crescendo diffusion and reordered equations , 2009, 0902.0076.

[5]  Marco Berghoff,et al.  Crystal-melt interface mobility in bcc Fe: Linking molecular dynamics to phase-field and phase-field crystal modeling , 2018 .

[6]  D. S. Kershaw,et al.  Flux limiting nature`s own way -- A new method for numerical solution of the transport equation , 1976 .

[7]  D. Mihalas,et al.  Foundations of Radiation Hydrodynamics , 1985 .

[8]  Raymond L. Murray,et al.  Nuclear reactor physics , 1957 .

[9]  Michael Herty,et al.  Models, numerical methods, and uncertainty quantification for radiation therapy , 2017 .

[10]  G. C. Pomraning.  The Equations of Radiation Hydrodynamics , 2005 .

[11]  Marco Berghoff,et al.  Massively Parallel Large-scale Multi-model Simulation of Tumor Development , 2019 .

[12]  Marco Berghoff,et al.  Massively Parallel Stencil Code Solver with Autonomous Adaptive Block Distribution , 2018, IEEE Transactions on Parallel and Distributed Systems.

[13]  G. C. Pomraning,et al.  Linear Transport Theory , 1967 .

[14]  Cory D. Hauck,et al.  Optimization and large scale computation of an entropy-based moment closure , 2015, J. Comput. Phys..

[15]  Edward W. Larsen,et al.  Tutorial: The Nature of Transport Calculations Used in Radiation Oncology , 1997 .

[16]  Gerhard Wellein,et al.  LIKWID: A Lightweight Performance-Oriented Tool Suite for x86 Multicore Environments , 2010, 2010 39th International Conference on Parallel Processing Workshops.

[17]  I. Müller,et al.  Rational Extended Thermodynamics , 1993 .

[18]  Benjamin Seibold,et al.  Asymptotic derivation and numerical investigation of time-dependent simplified PN equations , 2013, J. Comput. Phys..

[19]  Bingjing Su,et al.  Variable Eddington Factors and Flux Limiters in Radiative Transfer , 2001 .

[20]  B A Fraass,et al.  Electron dose calculations using the Method of Moments. , 1997, Medical physics.

[21]  Benjamin Seibold,et al.  StaRMAP---A Second Order Staggered Grid Method for Spherical Harmonics Moment Equations of Radiative Transfer , 2012, ACM Trans. Math. Softw..

[22]  Gordon L. Olson,et al.  Second-order time evolution of PN equations for radiation transport , 2009, J. Comput. Phys..

[23]  Edward W. Larsen,et al.  Asymptotic Derivation of the Multigroup P1 and Simplified PN Equations with Anisotropic Scattering , 1996 .

[24]  James Paul Holloway,et al.  On solutions to the Pn equations for thermal radiative transfer , 2008, J. Comput. Phys..

[25]  Benjamin Seibold,et al.  Optimal prediction for radiative transfer: A new perspective on moment closure , 2008, 0806.4707.

[26]  Anthony B. Davis,et al.  3D Radiative Transfer in Cloudy Atmospheres , 2005 .

[27]  M. Modest.  Radiative heat transfer , 1993 .

[28]  Eduardo F. D'Azevedo,et al.  MiniApps derived from production HPC applications using multiple programing models , 2018, Int. J. High Perform. Comput. Appl..

[29]  Jim E. Morel,et al.  Analysis of Ray-Effect Mitigation Techniques , 2003 .

[30]  Michael Herty,et al.  Optimal treatment planning in radiotherapy based on Boltzmann transport calculations , 2007 .

[31]  Marco Berghoff,et al.  Non-collective Scalable Global Network Based on Local Communications , 2018, 2018 IEEE/ACM 9th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (scalA).

[32]  Ryan G. McClarren,et al.  Positive PN Closures , 2010, SIAM J. Sci. Comput..

[33]  Michael Herty,et al.  Optimal Treatment Planning in Radiotherapy Based on Boltzmann Transport Equations , 2008 .

[34]  S. P. Gill,et al.  Physics of Shock Waves and High-Temperature Hydrodynamic Phenomena , 2002 .

[35]  Ryan G. McClarren,et al.  Semi-implicit time integration for PN thermal radiative transfer , 2008, J. Comput. Phys..

[36]  Samuel Williams,et al.  Roofline: an insightful visual performance model for multicore architectures , 2009, CACM.

[37]  C. D. Levermore,et al.  Relating Eddington factors to flux limiters , 1984 .

[38]  E. Richard Cohen,et al.  Neutron Transport Theory , 1959 .