Massively Parallel Algorithms for the Lattice Boltzmann Method on NonUniform Grids

The lattice Boltzmann method exhibits excellent scalability on current supercomputing systems and has thus increasingly become an alternative method for large-scale nonstationary flow simulations, reaching up to a trillion ($10^{12}$) grid nodes. Additionally, grid refinement can lead to substantial savings in memory and compute time. These savings, however, come at the cost of much more complex data structures and algorithms. In particular, the interface between subdomains with different grid sizes must receive special treatment. In this article, we present parallel algorithms, distributed data structures, and communication routines that are implemented in the software framework waLBerla in order to support large-scale, massively parallel lattice Boltzmann-based simulations on nonuniform grids. Additionally, we evaluate the performance of our approach on two current petascale supercomputers. On an IBM Blue Gene/Q system, the largest weak scaling benchmarks with refined grids are executed with almost 2 mi...

[1]  Martin Geier,et al.  Multi-thread implementations of the lattice Boltzmann method on non-uniform grids for CPUs and GPUs , 2011, Comput. Math. Appl..

[2]  B. Shizgal,et al.  Generalized Lattice-Boltzmann Equations , 1994 .

[3]  D. Kandhai,et al.  A generic, mass conservative local grid refinement technique for lattice‐Boltzmann schemes , 2006 .

[4]  Peter V. Coveney,et al.  Lattice-Boltzmann methods in fluid dynamics: Turbulence and complex colloidal fluids , 2011 .

[5]  L. Luo,et al.  Lattice Boltzmann Model for the Incompressible Navier–Stokes Equation , 1997 .

[6]  Manfred Krafczyk,et al.  A parallelisation concept for a multi-physics lattice Boltzmann prototype based on hierarchical grids , 2008 .

[7]  Tim Reis,et al.  The lattice Boltzmann method for complex flows , 2007 .

[8]  T. Neckel,et al.  A dynamic mesh refinement technique for Lattice Boltzmann simulations on octree-like grids , 2013 .

[9]  P. Bhatnagar,et al.  A Model for Collision Processes in Gases. I. Small Amplitude Processes in Charged and Neutral One-Component Systems , 1954 .

[10]  Amanda Elizabeth Randles,et al.  Modeling cardiovascular hemodynamics using the lattice Boltzmann method on massively parallel supercomputers , 2013 .

[11]  Hans-Joachim Bungartz,et al.  The PDE framework Peano applied to fluid dynamics: an efficient implementation of a parallel multiscale fluid dynamics solver on octree-like adaptive Cartesian grids , 2010 .

[12]  Orestis Malaspinas,et al.  Advances in multi-domain lattice Boltzmann grid refinement , 2012, J. Comput. Phys..

[13]  S. Mohanty,et al.  Drag correlation for dilute and moderately dense fluid-particle systems using the lattice Boltzmann method , 2014, 1401.2025.

[14]  Peter Bailey,et al.  Accelerating Lattice Boltzmann Fluid Flow Simulations Using Graphics Processors , 2009, 2009 International Conference on Parallel Processing.

[15]  Shiyi Chen,et al.  LATTICE BOLTZMANN METHOD FOR FLUID FLOWS , 2001 .

[16]  Ignacio Pagonabarraga,et al.  LUDWIG: A parallel Lattice-Boltzmann code for complex fluids , 2001 .

[17]  Mathias J. Krause,et al.  Hybrid Parallel Simulations of Fluid Flows in Complex Geometries: Application to the Human Lungs , 2010, Euro-Par Workshops.

[18]  T. Neckel,et al.  Navier–Stokes and Lattice–Boltzmann on octree‐like grids in the Peano framework , 2011 .

[19]  Peter Sanders,et al.  Optimized Hybrid Parallel Lattice Boltzmann Fluid Flow Simulations on Complex Geometries , 2012, Euro-Par.

[20]  Andy R. Terrel,et al.  ForestClaw: Hybrid forest-of-octrees AMR for hyperbolic conservation laws , 2013, PARCO.

[21]  D. d'Humières,et al.  Study of simple hydrodynamic solutions with the two-relaxation-times lattice Boltzmann scheme , 2008 .

[22]  Ulrich Rüde,et al.  Massively parallel phase-field simulations for ternary eutectic directional solidification , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[23]  W. Shyy,et al.  A multi‐block lattice Boltzmann method for viscous fluid flows , 2002 .

[24]  Peter M. A. Sloot,et al.  Finite-Difference Lattice-BGK methods on nested grids , 2000 .

[25]  Ulrich Rüde,et al.  Simulating fast electron beam melting with a parallel thermal free surface lattice Boltzmann method , 2014, Comput. Math. Appl..

[26]  M. Krafczyk,et al.  An adaptive scheme using hierarchical grids for lattice Boltzmann multi-phase flow simulations , 2006 .

[27]  Henning Meyerhenke,et al.  Dynamic Load Balancing for Parallel Numerical Simulations Based on Repartitioning with Disturbed Diffusion , 2009, 2009 15th International Conference on Parallel and Distributed Systems.

[28]  Y. Qian,et al.  Lattice BGK Models for Navier-Stokes Equation , 1992 .

[29]  Thomas Sauerwald,et al.  A new diffusion-based multilevel algorithm for computing graph partitions of very high quality , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[30]  D. d'Humières,et al.  Two-relaxation-time Lattice Boltzmann scheme: About parametrization, velocity, pressure and mixed boundary conditions , 2008 .

[31]  Ulrich Rüde,et al.  Parallel Simulations of Self-propelled Microorganisms , 2013, PARCO.

[32]  Ulrich Rüde,et al.  Lehrstuhl Für Informatik 10 (systemsimulation) Walberla: Hpc Software Design for Computational Engineering Simulations Walberla: Hpc Software Design for Computational Engineering Simulations , 2010 .

[33]  Gerhard Wellein,et al.  Exploring performance and power properties of modern multi‐core chips via simple machine models , 2012, Concurr. Comput. Pract. Exp..

[34]  Kannan Masilamani,et al.  Complex fluid simulations with the parallel tree-based Lattice Boltzmann solver Musubi , 2014, J. Comput. Sci..

[35]  Peter V. Coveney,et al.  Analysing and modelling the performance of the HemeLB lattice-Boltzmann simulation environment , 2012, J. Comput. Sci..

[36]  O. Filippova,et al.  Grid Refinement for Lattice-BGK Models , 1998 .

[37]  Sabine Roller,et al.  Multi-scale Lattice Boltzmann simulations on distributed octrees , 2014 .

[38]  Phillip Colella,et al.  Interpolation methods and the accuracy of lattice-Boltzmann mesh refinement , 2014, J. Comput. Phys..

[39]  Raoyang Zhang,et al.  Grid refinement in lattice Boltzmann methods based on volumetric formulation , 2006 .

[40]  Ulrich Rüde,et al.  Numerical simulation of adsorption and bubble interaction in protein foams using a lattice Boltzmann method. , 2014, Food & function.

[41]  D. d'Humières,et al.  Multiple–relaxation–time lattice Boltzmann models in three dimensions , 2002, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[42]  Stefan Becker,et al.  Flow-structure-acoustic interaction in a human voice model. , 2009, The Journal of the Acoustical Society of America.

[43]  Carsten Burstedde,et al.  p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees , 2011, SIAM J. Sci. Comput..

[44]  Ulrich Rüde,et al.  A framework for hybrid parallel flow simulations with a trillion cells in complex geometries , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[45]  Vincent Heuveline,et al.  THE OPENLB PROJECT: AN OPEN SOURCE AND OBJECT ORIENTED IMPLEMENTATION OF LATTICE BOLTZMANN METHODS , 2007 .

[46]  Ulrich Rüde,et al.  Parallel multiphysics simulations of charged particles in microfluidic flows , 2014, J. Comput. Sci..

[47]  Y. T. Chew,et al.  Application of multi-block approach in the immersed boundary-lattice Boltzmann method for viscous fluid flows , 2006, J. Comput. Phys..

[48]  Abdulmajeed A. Mohamad,et al.  A critical evaluation of force term in lattice Boltzmann method, natural convection problem , 2010 .

[49]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[50]  Mathias J. Krause,et al.  Towards a hybrid parallelization of lattice Boltzmann methods , 2009, Comput. Math. Appl..

[51]  Jonas Latt.  Choice of units in lattice Boltzmann simulations , 2008 .