A Python extension for the massively parallel multiphysics simulation framework waLBerla

We present a Python extension to the massively parallel HPC simulation toolkit waLBerla. waLBerla is a framework for stencil based algorithms operating on block-structured grids, with the main application field being fluid simulations in complex geometries using the lattice Boltzmann method. Careful performance engineering results in excellent node performance and good scalability to over 400,000 cores. To increase the usability and flexibility of the framework, a Python interface was developed. Python extensions are used at all stages of the simulation pipeline: they simplify and automate scenario setup, evaluation, and plotting. We show how our Python interface outperforms the existing text-file-based configuration mechanism, providing features like automatic nondimensionalization of physical quantities and handling of complex parameter dependencies. Furthermore, Python is used to process and evaluate results while the simulation is running, leading to smaller output files and the possibility to adjust parameters dependent on the current simulation state. C++ data structures are exported such that a seamless interfacing to other numerical Python libraries is possible. The expressive power of Python and the performance of C++ make development of efficient code with low time effort possible.

[1]  Matthew G. Knepley,et al.  PyClaw: Accessible, Extensible, Scalable Tools for Wave Propagation Problems , 2011, SIAM J. Sci. Comput..

[2]  J. Dongarra,et al.  The Impact of Multicore on Computational Science Software , 2007 .

[3]  Matthew G. Knepley,et al.  PetClaw: a scalable parallel nonlinear wave propagation solver for Python , 2011, SpringSim.

[4]  Ulrich Rüde,et al.  Massively parallel phase-field simulations for ternary eutectic directional solidification , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[5]  Amy Henderson Squilacote The Paraview Guide , 2008 .

[6]  Ulrich Rüde,et al.  Performance modeling and analysis of heterogeneous lattice Boltzmann simulations on CPU-GPU clusters , 2015, Parallel Comput..

[7]  David Joyner,et al.  Open source computer algebra systems: SymPy , 2012, ACCA.

[8]  Kyle T. Mandli,et al.  Using Python to Construct a Scalable Parallel Nonlinear Wave Solver , 2011 .

[9]  Anders Logg,et al.  Automated Solution of Differential Equations by the Finite Element Method: The FEniCS Book , 2012 .

[10]  Andy R. Terrel From Equations to Code: Automated Scientific Computing , 2011, Computing in Science & Engineering.

[11]  Michael Junk,et al.  On the relation between lattice variables and physical quantities in lattice Boltzmann simulations , 2006 .

[12]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[13]  Ulrich Rüde,et al.  Coupling multibody dynamics and computational fluid dynamics on 8192 processor cores , 2010, Parallel Comput..

[14]  Ulrich Rüde,et al.  Numerical investigations on hatching process strategies for powder-bed-based additive manufacturing using an electron beam , 2014, 1403.3251.

[15]  JoynerDavid,et al.  Open source computer algebra systems , 2012 .

[16]  Ulrich Rüde,et al.  Localized Parallel Algorithm for Bubble Coalescence in Free Surface Lattice-Boltzmann Method , 2009, Euro-Par.

[17]  Andy R. Terrel,et al.  ForestClaw: Hybrid forest-of-octrees AMR for hyperbolic conservation laws , 2013, PARCO.

[18]  Gaël Varoquaux,et al.  The NumPy Array: A Structure for Efficient Numerical Computation , 2011, Computing in Science & Engineering.

[19]  D. d'Humières,et al.  Study of simple hydrodynamic solutions with the two-relaxation-times lattice Boltzmann scheme , 2008 .

[20]  Ulrich Rüde,et al.  A framework for hybrid parallel flow simulations with a trillion cells in complex geometries , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[21]  Tamara G. Kolda,et al.  An overview of the Trilinos project , 2005, TOMS.

[22]  Anders Logg,et al.  A compiler for variational forms , 2006, TOMS.

[23]  Ulrich Rüde,et al.  Lehrstuhl Für Informatik 10 (systemsimulation) Walberla: Hpc Software Design for Computational Engineering Simulations Walberla: Hpc Software Design for Computational Engineering Simulations , 2010 .

[24]  Michael A. Heroux,et al.  PyTrilinos: High-performance distributed-memory solvers for Python , 2006, TOMS.

[25]  Cyrus K. Aidun,et al.  Lattice-Boltzmann Method for Complex Flows , 2010 .

[26]  U. Rüde,et al.  Lattice Boltzmann Model for Free Surface Flow for Modeling Foaming , 2005 .

[27]  Ulrich Rüde,et al.  Massively Parallel Algorithms for the Lattice Boltzmann Method on NonUniform Grids , 2015, SIAM J. Sci. Comput..

[28]  M. Januszewski,et al.  Sailfish: A flexible multi-GPU implementation of the lattice Boltzmann method , 2013, Comput. Phys. Commun..

[29]  Y. Qian,et al.  Lattice BGK Models for Navier-Stokes Equation , 1992 .

[30]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[31]  D. d'Humières,et al.  Two-relaxation-time Lattice Boltzmann scheme: About parametrization, velocity, pressure and mixed boundary conditions , 2008 .

[32]  Ulrich Rüde,et al.  Large scale simulation of fluid structure interaction using Lattice Boltzmann methods and the `physics engine' , 2008 .