EMPIRE-PIC: A Performance Portable Unstructured Particle-in-Cell Code

In this paper we introduce EMPIRE-PIC, a finite element method particlein-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in twoand three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems – a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel’s Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs. AMS subject classifications: To be provided by authors

[1]  Qing Zhang,et al.  The Gyrokinetic Particle Simulation of Fusion Plasmas on Tianhe-2 Supercomputer , 2016, 2016 7th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (ScalA).

[2]  R. Marchand,et al.  PTetra, a Tool to Simulate Low Orbit Satellite–Plasma Interaction , 2012, IEEE Transactions on Plasma Science.

[3]  Robert L. Lee,et al.  Don''t suppress the wiggles|they''re telling you something! Computers and Fluids , 1981 .

[4]  Benjamin Bergen,et al.  0.374 Pflop/s trillion-particle kinetic modeling of laser plasma interaction on roadrunner , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[5]  R W Hockney,et al.  Computer Simulation Using Particles , 1966 .

[6]  Alex Friedman,et al.  A second-order implicit particle mover with adjustable damping , 1990 .

[7]  Samuel Williams,et al.  Kinetic turbulence simulations at extreme scale on leadership-class systems , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[8]  Phillip Colella,et al.  Application of adaptive mesh refinement to particle-in-cell simulations of plasmas and beams , 2003 .

[9]  B. M. Marder,et al.  A method for incorporating Gauss' lasw into electromagnetic pic codes , 1987 .

[10]  D. R. Nicholson.  Introduction to Plasma Theory , 1983 .

[11]  A. Bruce Langdon,et al.  THEORY OF PLASMA SIMULATION USING FINITE-SIZE PARTICLES. , 1970 .

[12]  Carretera de Valencia,et al.  The finite element method in electromagnetics , 2000 .

[13]  D. Payan,et al.  SPIS Open-Source Code: Methods, Capabilities, Achievements, and Prospects , 2008, IEEE Transactions on Plasma Science.

[14]  K. Bowers,et al.  Ultrahigh performance three-dimensional electromagnetic relativistic kinetic plasma simulationa) , 2008 .

[15]  R. G. Evans,et al.  Contemporary particle-in-cell approach to laser-plasma modelling , 2015 .

[16]  Robert Hager,et al.  A new hybrid-Lagrangian numerical scheme for gyrokinetic simulation of tokamak edge plasma , 2016, J. Comput. Phys..

[17]  Luis Chacón,et al.  An efficient mixed-precision, hybrid CPU-GPU implementation of a nonlinearly implicit one-dimensional particle-in-cell algorithm , 2011, J. Comput. Phys..

[18]  R. W. Hockney,et al.  COMPUTER EXPERIMENT OF ANOMALOUS DIFFUSION , 1966 .

[19]  Iu. L. Klimontovich,et al.  The statistical theory of non-equilibrium processes in a plasma , 1967 .

[20]  Daniel Sunderland,et al.  Kokkos: Enabling manycore performance portability through polymorphic memory access patterns , 2014, J. Parallel Distributed Comput..

[21]  Francesca Rapetti,et al.  Whitney Forms of Higher Degree , 2009, SIAM J. Numer. Anal..

[22]  Jonathan Joseph Hu,et al.  MueLu User?s Guide. , 2019 .

[23]  Ümit V. Çatalyürek,et al.  The Zoltan and Isorropia parallel toolkits for combinatorial scientific computing: Partitioning, ordering and coloring , 2012, Sci. Program..

[24]  Wei Lu,et al.  OSIRIS: A Three-Dimensional, Fully Relativistic Particle in Cell Code for Modeling Plasma Based Accelerators , 2002, International Conference on Computational Science.

[25]  Stephen A. Jarvis,et al.  Higher-order particle representation for particle-in-cell simulations , 2021, J. Comput. Phys..

[26]  E. Quataert,et al.  PARTICLE-IN-CELL SIMULATIONS OF CONTINUOUSLY DRIVEN MIRROR AND ION CYCLOTRON INSTABILITIES IN HIGH BETA ASTROPHYSICAL AND HELIOSPHERIC PLASMAS , 2014, 1402.0014.

[27]  Viktor K. Decyk,et al.  Adaptable Particle-in-Cell algorithms for graphical processing units , 2010, Comput. Phys. Commun..

[28]  A. Bruce Langdon,et al.  On enforcing Gauss' law in electromagnetic particle-in-cell codes , 1992 .

[29]  Charles K. Birdsall,et al.  Clouds-in-Clouds, Clouds-in-Cells Physics for Many-Body Plasma Simulation , 1997 .

[30]  K. Yee.  Numerical solution of initial boundary value problems involving maxwell's equations in isotropic media , 1966 .

[31]  G. E. Sasser,et al.  3-D ICEPIC simulations of the relativistic klystron oscillator , 2000 .

[32]  Thomas L. Sterling,et al.  BEOWULF: A Parallel Workstation for Scientific Computation , 1995, ICPP.

[33]  Pavel B. Bochev,et al.  An Algebraic Multigrid Approach Based on a Compatible Gauge Reformulation of Maxwell's Equations , 2008, SIAM J. Sci. Comput..

[34]  Courtenay T. Vaughan,et al.  ASC Tri-lab Co-design Level 2 Milestone Report 2015 , 2015 .

[35]  Stephen A. Jarvis,et al.  Performance Optimisation of Inertial Confinement Fusion Codes using Mini-applications , 2018, Int. J. High Perform. Comput. Appl..

[36]  Gregory Fridman,et al.  Applied Plasma Medicine , 2008 .

[37]  Jean E. Roberts,et al.  Mixed and hybrid finite element methods , 1987 .

[38]  Francis F. Chen,et al.  Introduction to Plasma Physics and Controlled Fusion , 2015 .

[39]  Stephen A. Jarvis,et al.  Towards a portable and future-proof particle-in-cell plasma physics code , 2013 .

[40]  Stephen A. Jarvis,et al.  Performance of a Second Order Electrostatic Particle-in-Cell Algorithm on Modern Many-Core Architectures , 2018, UKPEW.

[41]  H Burau,et al.  PIConGPU: A Fully Relativistic Particle-in-Cell Code for a GPU Cluster , 2010, IEEE Transactions on Plasma Science.

[42]  Victor W. Lee,et al.  Implications of a metric for performance portability , 2017, Future Gener. Comput. Syst..

[43]  John D. Villasenor,et al.  Rigorous charge conservation for local electromagnetic field solvers , 1992 .

[44]  Liang Wang,et al.  The Plasma Simulation Code: A modern particle-in-cell code with patch-based load-balancing , 2016, J. Comput. Phys..

[45]  John P. Verboncoeur,et al.  Conformal Electromagnetic Particle in Cell: A Review , 2015, IEEE Transactions on Plasma Science.

[46]  R. Hiptmair.  Finite elements in computational electromagnetism , 2002, Acta Numerica.

[47]  Stephane Ethier,et al.  Performance of particle in cell methods on highly concurrent computational architectures , 2007 .

[48]  Pavel B. Bochev,et al.  Principles of Mimetic Discretizations of Differential Operators , 2006 .

[49]  J. Dawson.  Particle simulation of plasmas , 1983 .

[50]  Pavel B. Bochev,et al.  Solving PDEs with Intrepid , 2012, Sci. Program..

[51]  Sivasankaran Rajamanickam,et al.  Amesos2 and Belos: Direct and iterative solvers for large sparse linear systems , 2012, Sci. Program..

[52]  W. Zhang,et al.  Warp-X: A new exascale computing platform for beam–plasma simulations , 2017, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment.

[53]  C. Birdsall,et al.  Plasma Physics via Computer Simulation , 2018 .

[54]  Martin Campos Pinto,et al.  Handling the divergence constraints in Maxwell and Vlasov-Maxwell simulations , 2016, Appl. Math. Comput..

[55]  A. Bossavit.  A rationale for 'edge-elements' in 3-D fields computations , 1988 .

[56]  J. Nédélec.  Mixed finite elements in ℝ3 , 1980 .

[57]  Tamara G. Kolda,et al.  An overview of the Trilinos project , 2005, TOMS.

[58]  T. H. Dupree.  KINETIC THEORY OF PLASMA AND THE ELECTROMAGNETIC FIELD , 1963 .

[59]  John N. Shadid,et al.  IMEX and exact sequence discretization of the multi-fluid plasma model , 2019, J. Comput. Phys..

[60]  Leonid Oliker,et al.  Extreme Scale Plasma Turbulence Simulations on Top Supercomputers Worldwide , 2016, SC16: International Conference for High Performance Computing, Networking, Storage and Analysis.

[61]  T. D. Pointon.  Second-order, exact charge conservation for electromagnetic particle-in-cell simulation in complex geometry , 2008, Comput. Phys. Commun..