On architecture and performance of adaptive mesh refinement in an electrostatics Particle-In-Cell code

This article presents a hardware architecture independent implementation of an adaptive mesh refinement Poisson solver that is integrated into the electrostatic Particle-In-Cell beam dynamics code OPAL. The Poisson solver is solely based on second generation Trilinos packages to ensure the desired hardware portability. Based on the massively parallel framework AMReX, formerly known as BoxLib, the new adaptive mesh refinement interface provides several refinement policies in order to enable precise large-scale neighbouring bunch simulations in high intensity cyclotrons. The solver is validated with a built-in multigrid solver of AMReX and a test problem with analytical solution. The parallel scalability is presented as well as an example of a neighbouring bunch simulation that covers the scale of the later anticipated physics simulation.

[1]  A. Adelmann,et al.  Beam dynamics in high intensity cyclotrons including neighboring bunch effects: Model, implementation, and application , 2010, 1003.0326.

[2]  Jeffrey A. F. Hittinger,et al.  Block-structured adaptive mesh refinement algorithms for Vlasov simulation , 2011, J. Comput. Phys..

[3]  M. Berger,et al.  Adaptive mesh refinement for hyperbolic partial differential equations , 1982 .

[4]  M. Norman,et al.  yt: A MULTI-CODE ANALYSIS TOOLKIT FOR ASTROPHYSICAL SIMULATION DATA , 2010, 1011.3514.

[5]  W. Zhang,et al.  Warp-X: A new exascale computing platform for beam–plasma simulations , 2017, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment.

[6]  Daniel F. Martin,et al.  Solving Poisson's Equation using Adaptive Mesh Renemen t , 1996 .

[7]  Jonathan Joseph Hu,et al.  MueLu User?s Guide. , 2019 .

[8]  R. Gordon,et al.  A finite difference approach that employs an asymptotic boundary condition on a rectangular outer boundary for modeling two-dimensional transmission line structures , 1993 .

[9]  A. Bayliss,et al.  Radiation boundary conditions for wave-like equations , 1980 .

[10]  Daniel Sunderland,et al.  Kokkos: Enabling manycore performance portability through polymorphic memory access patterns , 2014, J. Parallel Distributed Comput..

[11]  Jean-Luc Vay,et al.  Novel methods in the Particle-In-Cell accelerator Code-Framework Warp , 2012 .

[12]  Phillip Colella,et al.  A 4th-Order Particle-in-Cell Method with Phase-Space Remapping for the Vlasov-Poisson Equation , 2016, SIAM J. Sci. Comput..

[13]  Katharina Kormann,et al.  A Semi-Lagrangian Vlasov Solver in Tensor Train Format , 2014, SIAM J. Sci. Comput..

[14]  Vladimir I Kolobov,et al.  Electrostatic PIC with adaptive Cartesian mesh , 2016 .

[15]  Daniel Sunderland,et al.  Manycore performance-portability: Kokkos multidimensional array library , 2012 .

[16]  U. Shumlak,et al.  Physics-Based-Adaptive Plasma Model for High-Fidelity Numerical Simulations , 2018, Front. Phys..

[17]  M. Gunzburger,et al.  Boundary conditions for the numerical solution of elliptic equations in exterior regions , 1982 .

[18]  K. Crandall,et al.  Beam Halo in Proton Linac Beams , 2000 .

[19]  A. Adelmann,et al.  Evolution of a beam dynamics model for the transport line in a proton therapy facility , 2017, 1705.10549.

[20]  B. M. Fulk.  MATH , 1992 .

[21]  Heidi K. Thornquist,et al.  Amesos2 and Belos: Direct and iterative solvers for large sparse linear systems , 2012 .

[22]  A. Adelmann,et al.  OPAL a Versatile Tool for Charged Particle Accelerator Simulations , 2019, 1905.06654.

[23]  D. A. Dunnett.  Classical Electrodynamics , 2020, Nature.

[24]  Peter Arbenz,et al.  A novel adaptive time stepping variant of the Boris-Buneman integrator for the simulation of particle accelerators with space charge , 2012, J. Comput. Phys..

[25]  P. Colella,et al.  Local adaptive mesh refinement for shock hydrodynamics , 1989 .

[26]  Michael A. Heroux,et al.  Tpetra, and the use of generic programming in scientific computing , 2012, Sci. Program..

[27]  Raghwendra Kumar,et al.  Boundary conditions for the solution of the three-dimensional Poisson equation in open metallic enclosures , 2015, 1510.04400.

[28]  R. Mittra,et al.  Asymptotic boundary conditions for finite element analysis of three-dimensional transmission line discontinuities , 1990 .

[29]  P. Colella,et al.  A Conservative Adaptive Projection Method for the Variable Density Incompressible Navier-Stokes Equations , 1998 .

[30]  R W Hockney,et al.  Computer Simulation Using Particles , 1966 .

[31]  Peter Arbenz,et al.  A fast parallel Poisson solver on irregular domains applied to beam dynamics simulations , 2009, J. Comput. Phys..

[32]  Uldis Locans,et al.  The Dynamic Kernel Scheduler - Part 1 , 2016, Comput. Phys. Commun..

[33]  Daniel Sunderland,et al.  Manycore performance-portability: Kokkos multidimensional array library , 2012, Sci. Program..

[34]  Phillip Colella,et al.  Controlling self-force errors at refinement boundaries for AMR-PIC , 2010, J. Comput. Phys..

[35]  W. C. Swope,et al.  A computer simulation method for the calculation of equilibrium constants for the formation of physi , 1981 .