On the accuracy and performance of the lattice Boltzmann method with 64-bit, 32-bit and novel 16-bit number formats

Fluid dynamics simulations with the lattice Boltzmann method (LBM) are very memory-intensive. Alongside reduction in memory footprint, significant performance benefits can be achieved by using FP32 (single) precision compared to FP64 (double) precision, especially on GPUs. Here, we evaluate the possibility to use even FP16 and Posit16 (half) precision for storing fluid populations, while still carrying arithmetic operations in FP32. For this, we first show that the commonly occurring number range in the LBM is a lot smaller than the FP16 number range. Based on this observation, we develop novel 16-bit formats – based on a modified IEEE-754 and on a modified Posit standard – that are specifically tailored to the needs of the LBM. We then carry out an in-depth characterization of LBM accuracy for six different test systems with increasing complexity: Poiseuille flow, Taylor-Green vortices, Karman vortex streets, liddriven cavity, a microcapsule in shear flow (utilizing the immersed-boundary method) and finally the impact of a raindrop (based on a Volume-of-Fluid approach). We find that the difference in accuracy between FP64 and FP32 is negligible in almost all cases, and that for a large number of cases even 16-bit is sufficient. Finally, we provide a detailed performance analysis of all precision levels on a large number of hardware microarchitectures and show that significant speedup is achieved with mixed FP32/16-bit.

[1]  Louis A. Povinelli,et al.  Large-scale computation of incompressible viscous flow by least-squares finite element method , 1994 .

[2]  Ivo Petráš,et al.  Cross-Platform GPU-Based Implementation of Lattice Boltzmann Method Solver Using ArrayFire Library , 2021, Mathematics.

[3]  Introduction to the immersed boundary method 1 Motivation 1 . 1 Hydrodynamics and boundary conditions , 2022 .

[4]  Massimo Bernaschi,et al.  Towards Exascale Lattice Boltzmann computing , 2019, Computers & Fluids.

[5]  G. Boroni,et al.  FULL GPU Implementation of Lattice-Boltzmann Methods with Immersed Boundary Conditions for Fast Fluid Simulations , 2017 .

[6]  Ge Wei,et al.  Efficient parallel implementation of the lattice Boltzmann method on large clusters of graphic processing units , 2012 .

[7]  J. Harting,et al.  SWALBE: A lattice Boltzmann solver of the shallow water equations for thin liquid film flows , 2018, 1810.07642.

[8]  G. Batchelor,et al.  An Introduction to Fluid Dynamics , 1968 .

[9]  Seyong Lee,et al.  GPU Data Access on Complex Geometries for D3Q19 Lattice Boltzmann Method , 2018, 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[10]  Hans-Joachim Bungartz,et al.  A Holistic Scalable Implementation Approach of the Lattice Boltzmann Method for CPU/GPU Heterogeneous Clusters , 2017, Comput..

[11]  Fabian Klemens,et al.  OpenLB - Open source lattice Boltzmann code , 2020, Comput. Math. Appl..

[12]  D. d'Humières,et al.  Multiple–relaxation–time lattice Boltzmann models in three dimensions , 2002, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[13]  Bernard Tourancheau,et al.  Author Manuscript, Published in "computers and Mathematics with Applications (2010)" a New Approach to the Lattice Boltzmann Method for Graphics Processing Units , 2011 .

[14]  Li-Min Huang,et al.  Scalable multi-relaxation-time lattice Boltzmann simulations on multi-GPU cluster , 2015 .

[15]  John L. Gustafson,et al.  Beating Floating Point at its Own Game: Posit Arithmetic , 2017, Supercomput. Front. Innov..

[16]  Moritz Lehmann High Performance Free Surface LBM on GPUs , 2021 .

[17]  Gerhard Wellein,et al.  On the single processor performance of simple lattice Boltzmann kernels , 2006 .

[18]  Peter Bailey,et al.  Accelerating geoscience and engineering system simulations on graphics hardware , 2009, Comput. Geosci..

[19]  H. Yoshida,et al.  Locally mesh-refined lattice Boltzmann method for fuel debris air cooling analysis on GPU supercomputer , 2020 .

[20]  Gudrun Thäter,et al.  Auto-vectorization friendly parallel lattice Boltzmann streaming scheme for direct addressing , 2019, Computers & Fluids.

[21]  Raffaele Tripiccione,et al.  Massively parallel lattice-Boltzmann codes on large GPU clusters , 2016, Parallel Comput..

[22]  Markus Wittmann,et al.  Hardware-effiziente, hochparallele Implementierungen von Lattice-Boltzmann-Verfahren für komplexe Geometrien , 2016 .

[23]  Mathias J. Krause,et al.  Fluid Flow Simulation and Optimisation with Lattice Boltzmann Methods on High Performance Computers - Application to the Human Respiratory System , 2010 .

[24]  Mathias J. Krause,et al.  OpenLB : towards an efficient parallel open source library for lattice Boltzmann fluid flow simulations , 2008 .

[25]  Alistair J. Revell,et al.  Parallelisation of an interactive lattice-Boltzmann method on an Android-powered mobile device , 2017, Adv. Eng. Softw..

[26]  Alireza Entezari,et al.  The Lattice-Boltzmann Method on Optimal Sampling Lattices , 2009, IEEE Transactions on Visualization and Computer Graphics.

[27]  Nicholas J. Higham,et al.  Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems , 2020, Proceedings of the Royal Society A.

[28]  Jonathan B. Jeppener-Haltenhoff,et al.  OpenLB Release 1.4: Open Source Lattice Boltzmann Code , 2020 .

[29]  Hartwig Anzt,et al.  A Modular Precision Format for Decoupling Arithmetic Format and Storage Format , 2018, Euro-Par Workshops.

[30]  Dominique Barthès-Biesel,et al.  Motion and Deformation of Elastic Capsules and Vesicles in Flow , 2016 .

[31]  Skordos,et al.  Initial and boundary conditions for the lattice Boltzmann method. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[32]  S. Gekle,et al.  Theory and algorithms to compute Helfrich bending forces: a review , 2017, Journal of physics. Condensed matter : an Institute of Physics journal.

[33]  A. Salsac,et al.  Flow-Induced Deformation of Artificial Capsules , 2010 .

[34]  Stephan Gekle,et al.  On the bending algorithms for soft objects in flows , 2016, Comput. Phys. Commun..

[35]  L. Luo,et al.  Lattice Boltzmann Model for the Incompressible Navier–Stokes Equation , 1997 .

[36]  Samuel Williams,et al.  Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures , 2008 .

[37]  Miloš Racković,et al.  Implementation of the Lattice Boltzmann Method on Heterogeneous Hardware and Platforms using OpenCL , 2012 .

[38]  J. Harting,et al.  Lattice Boltzmann simulations of stochastic thin film dewetting. , 2020, Physical review. E.

[39]  Peter Dueben,et al.  Compressing atmospheric data into its real information content , 2021, Nature Computational Science.

[40]  J. Harting,et al.  Mesoscopic electrohydrodynamic simulations of binary colloidal suspensions. , 2017, The Journal of chemical physics.

[41]  D. Wolf,et al.  Recent advances in the simulation of particle-laden flows , 2014, The European Physical Journal Special Topics.

[42]  Peter Bailey,et al.  Accelerating Lattice Boltzmann Fluid Flow Simulations Using Graphics Processors , 2009, 2009 International Conference on Parallel Processing.

[43]  Gerhard Wellein,et al.  Towards Optimal Performance for Lattice Boltzmann Applications on Terascale Computers , 2006 .

[44]  Nhat-Phuong Tran,et al.  Performance Optimization of 3D Lattice Boltzmann Flow Solver on a GPU , 2017, Sci. Program..

[45]  Takayuki Aoki,et al.  Multi-GPU performance of incompressible flow computation by lattice Boltzmann method on GPU cluster , 2011, Parallel Comput..

[46]  Bastien Chopard,et al.  Bridging the computational gap between mesoscopic and continuum modeling of red blood cells for fully resolved blood flow , 2019, J. Comput. Phys..

[47]  G. Taylor,et al.  Mechanism of the production of small eddies from large ones , 1937 .

[48]  Hans-Joachim Bungartz,et al.  International Conference on Computational Science , ICCS 2011 Free-Surface Lattice-Boltzmann Simulation on Many-Core Architectures , 2011 .

[49]  Peter V. Coveney,et al.  LB3D: A parallel implementation of the Lattice-Boltzmann method for simulation of interacting amphiphilic fluids , 2017, Comput. Phys. Commun..

[50]  F. Auricchio,et al.  Basis of the Lattice Boltzmann Method for Additive Manufacturing , 2020, Archives of Computational Methods in Engineering.

[51]  Raffaele Tripiccione,et al.  Optimizing communications in multi-GPU Lattice Boltzmann simulations , 2015, 2015 International Conference on High Performance Computing & Simulation (HPCS).

[52]  S. Gekle,et al.  Ejection of marine microplastics by raindrops: a computational and experimental study , 2021, Microplastics and Nanoplastics.

[53]  Matthew Chantry,et al.  Accelerating High-Resolution Weather Models with Deep-Learning Hardware , 2019, PASC.

[54]  Massimo Bernaschi,et al.  LBsoft: A parallel open-source software for simulation of colloidal systems , 2020, Comput. Phys. Commun..

[55]  B. Kaoui,et al.  Numerical simulations of complex fluid-fluid interface dynamics , 2012, 1208.2539.

[56]  Gerhard Wellein,et al.  Performance engineering for the Lattice Boltzmann method on GPGPUs: Architectural requirements and performance results , 2011, ArXiv.

[57]  Christophe Renaud,et al.  Accelerating Physical Simulations from a Multicomponent Lattice Boltzmann Method on a Single-Node Multi-GPU Architecture , 2015, 2015 10th International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC).

[58]  T. Neckel,et al.  A dynamic mesh refinement technique for Lattice Boltzmann simulations on octree-like grids , 2013 .

[59]  T. Kármán,et al.  Ueber den Mechanismus des Widerstandes, den ein bewegter Körper in einer Flüssigkeit erfährt , 1911 .

[60]  Zhangrong Qin,et al.  An Efficient Graphics Processing Unit Scheme for Complex Geometry Simulations Using the Lattice Boltzmann Method , 2020, IEEE Access.

[61]  C. J. Noakes,et al.  Optimized implementation of the Lattice Boltzmann Method on a graphics processing unit towards real-time fluid simulation , 2014, Comput. Math. Appl..

[62]  A. Ladd Numerical simulations of particulate suspensions via a discretized Boltzmann equation. Part 1. Theoretical foundation , 1993, Journal of Fluid Mechanics.

[63]  A. Purqon,et al.  Accuracy and Numerical Stabilty Analysis of Lattice Boltzmann Method with Multiple Relaxation Time for Incompressible Flows , 2017 .

[64]  J. Dongarra,et al.  Exploiting the Performance of 32 bit Floating Point Arithmetic in Obtaining 64 bit Accuracy (Revisiting Iterative Refinement for Linear Systems) , 2006, ACM/IEEE SC 2006 Conference (SC'06).

[65]  B. Shi,et al.  Discrete lattice effects on the forcing term in the lattice Boltzmann method. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[66]  Michael Griebel,et al.  Meshfree Methods for Partial Differential Equations IV , 2005 .

[67]  Christian Obrecht,et al.  LBM based flow simulation using GPU computing processor , 2010, Comput. Math. Appl..

[68]  Gerhard Wellein,et al.  Comparison of different propagation steps for lattice Boltzmann methods , 2011, Comput. Math. Appl..

[69]  Jean-Michel Muller,et al.  Posits: the good, the bad and the ugly , 2019, CoNGA'19.

[70]  Orestis Malaspinas,et al.  Parallel performance of an IB-LBM suspension simulation framework , 2015, J. Comput. Sci..

[71]  J. Harting,et al.  Interplay of inertia and deformability on rheological properties of a suspension of capsules , 2013, Journal of Fluid Mechanics.

[72]  Jonas Latt,et al.  Toward Full GPU Implementation of Fluid-Structure Interaction , 2019, 2019 18th International Symposium on Parallel and Distributed Computing (ISPDC).

[73]  Stephan Gekle,et al.  Analytic Solution to the Piecewise Linear Interface Construction Problem and its Application in Curvature Calculation for Volume-of-Fluid Simulation Codes , 2020, Comput..

[74]  Gerhard Wellein,et al.  Performance analysis and optimization strategies for a D3Q19 lattice Boltzmann kernel on nVIDIA GPUs using CUDA , 2011, Adv. Eng. Softw..

[75]  Alistair J. Revell,et al.  Memory transfer optimization for a lattice Boltzmann solver on Kepler architecture nVidia GPUs , 2013, Comput. Phys. Commun..

[76]  Ulrich Rüde,et al.  A flexible Patch-based lattice Boltzmann parallelization approach for heterogeneous GPU-CPU clusters , 2010, Parallel Comput..

[77]  Anne C. Elster,et al.  Porous Rock Simulations and Lattice Boltzmann on GPUs , 2009, PARCO.

[78]  Manfred Krafczyk,et al.  TeraFLOP computing on a desktop PC with GPUs for 3D CFD , 2008 .

[79]  R. Benzi,et al.  The lattice Boltzmann equation: theory and applications , 1992 .

[80]  Benoît Dupont de Dinechin,et al.  Improving 3D lattice boltzmann method stencil with asynchronous transfers on many-core processors , 2017, 2017 IEEE 36th International Performance Computing and Communications Conference (IPCCC).

[81]  Enrique S. Quintana-Ortí,et al.  Toward a modular precision ecosystem for high-performance computing , 2019, Int. J. High Perform. Comput. Appl..

[82]  Hans-Jörg Limbach,et al.  ESPResSo - an extensible simulation package for research on soft matter systems , 2006, Comput. Phys. Commun..

[83]  Marcelo J. Vénere,et al.  A Lattice-Boltzmann solver for 3D fluid simulation on GPU , 2012, Simul. Model. Pract. Theory.

[84]  Bernard Tourancheau,et al.  Global Memory Access Modelling for Efficient Implementation of the Lattice Boltzmann Method on Graphics Processing Units , 2010, VECPAR.

[85]  Erik W Draeger,et al.  Multi-GPU immersed boundary method hemodynamics simulations , 2020, J. Comput. Sci..

[86]  Bastien Chopard,et al.  Palabos-npFEM: Software for the Simulation of Cellular Blood Flow (Digital Blood) , 2020, Journal of Open Research Software.

[87]  T. G. Cowling,et al.  The mathematical theory of non-uniform gases : an account of the kinetic theory of viscosity, thermal conduction, and diffusion in gases , 1954 .

[88]  I. Tiselj,et al.  Lattice Boltzmann Method , 2022, Advanced Computational Techniques for Heat and Mass Transfer in Food Processing.

[89]  Orestis Malaspinas,et al.  Palabos: Parallel Lattice Boltzmann Solver , 2020, Comput. Math. Appl..

[90]  Philipp Neumann,et al.  Lattice Boltzmann Flow Simulation on Android Devices for Interactive Mobile-Based Learning , 2016, Euro-Par Workshops.

[91]  Martin Bauer,et al.  Highly efficient lattice Boltzmann multiphase simulations of immiscible fluids at high-density ratios on CPUs and GPUs through code generation , 2020, Int. J. High Perform. Comput. Appl..

[92]  M. Porfiri,et al.  Extreme flow simulations reveal skeletal adaptations of deep-sea sponges , 2021, Nature.

[93]  Stefan Turek,et al.  Lattice-Boltzmann Simulation of the Shallow-Water Equations with Fluid-Structure Interaction on Multi- and Manycore Processors , 2010, Facing the Multicore-Challenge.

[94]  C. Bogner,et al.  Tracing the horizontal transport of microplastics on rough surfaces , 2021, Microplastics and Nanoplastics.

[95]  Jonas Latt,et al.  Efficient LBM on GPUs for dense moving objects using immersed boundary condition , 2019, ArXiv.

[96]  Norberto Fueyo,et al.  Analysis of open boundary effects in unsteady lattice Boltzmann simulations , 2009, Comput. Math. Appl..

[97]  Andreas Lintermann,et al.  Lattice–Boltzmann simulations for complex geometries on high-performance computers , 2020, CEAS Aeronautical Journal.

[98]  Bernard Tourancheau,et al.  Multi-GPU implementation of the lattice Boltzmann method , 2013, Comput. Math. Appl..

[99]  Edo S. Boek,et al.  Enhancing Computational Precision for Lattice Boltzmann Schemes in Porous Media Flows , 2016, Comput..

[100]  Jaw-Yen Yang,et al.  Implicit Weighted ENO Schemes for the Three-Dimensional Incompressible Navier-Stokes Equations , 1998 .