Zonal Flow Solver (ZFS): a highly efficient multi-physics simulation framework

Multi-physics simulations are at the heart of today's engineering applications. The trend is towards more realistic and detailed simulations, which demand highly resolved spatial and temporal scales of various physical mechanisms to solve engineering problems in a reasonable amount of time. As a consequence, numerical codes need to run efficiently on high-performance computers. Therefore, the framework Zonal Flow Solver (ZFS) featuring lattice-Boltzmann, finite-volume, discontinuous Galerkin, level set and Lagrange solvers has been developed. The solvers can be combined to simulate, e.g. quasi-incompressible and compressible flow, aeroacoustics, moving boundaries and particle dynamics. In this manuscript, the multi-physics implementation of the coupling mechanisms are presented. The parallelisation approach, the involved solvers and their scalability on state-of-the-art heterogeneous high-performance computers are discussed. Various multi-physics applications complement the discussion. The results show ZFS to be a highly efficient and flexible multi-purpose tool that can be used to solve varying classes of coupled problems.

[1]  Andreas Lintermann,et al.  Fluid mechanics based classification of the respiratory efficiency of several nasal cavities , 2013, Comput. Biol. Medicine.

[2]  Andreas Lintermann,et al.  Performance of ODROID-MC1 for scientific flow problems , 2019, Future Gener. Comput. Syst..

[3]  Andreas Lintermann,et al.  Large-Scale Simulations of a Non-generic Helicopter Engine Nozzle and a Ducted Axial Fan , 2016 .

[4]  M. Meinke,et al.  Hydrodynamic instability and shear layer effect on the response of an acoustically excited laminar premixed flame , 2015 .

[5]  E. Krause,et al.  A comparison of second- and sixth-order methods for large-eddy simulations , 2002 .

[6]  W. Schröder,et al.  Acoustic perturbation equations based on flow decomposition via source filtering , 2003 .

[7]  Michael M. Resch,et al.  High Performance Computing on Vector Systems 2011 , 2012 .

[8]  B. Chopard,et al.  Theory and applications of an alternative lattice Boltzmann grid refinement algorithm. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[9]  Andreas Lintermann,et al.  CFD/CAA Simulations on HPC Systems , 2016 .

[10]  Jeffrey S. Vetter,et al.  Exploiting Lustre File Joining for Effective Collective IO , 2007, Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07).

[11]  M. Meinke,et al.  Nonlinear analysis of an acoustically excited laminar premixed flame , 2016 .

[12]  Wolfgang Schröder,et al.  Differential equation based constrained reinitialization for level set methods , 2008, J. Comput. Phys..

[13]  Zhuang Fengqing,et al.  Patients’ Responsibilities in Medical Ethics , 2016 .

[14]  B. V. Leer,et al.  Towards the ultimate conservative difference scheme V. A second-order sequel to Godunov's method , 1979 .

[15]  Steven G. Johnson,et al.  The Design and Implementation of FFTW3 , 2005, Proceedings of the IEEE.

[16]  Andreas Lintermann,et al.  Numerical Simulation of Nasal Cavity Flow Based on a Lattice-Boltzmann Method , 2010 .

[17]  Matthias Meinke,et al.  Aeroacoustic Analysis of a Helicopter Engine Jet Including a Realistic Nozzle Geometry , 2015 .

[18]  Michael A. Heroux.  Software Challenges for Extreme Scale Computing: Going From Petascale to Exascale Systems , 2009, Int. J. High Perform. Comput. Appl..

[19]  Matthias Meinke,et al.  Dynamic load balancing for direct-coupled multiphysics simulations , 2020 .

[20]  Jutta Docter,et al.  JUQUEEN: IBM Blue Gene/Q® Supercomputer System at the Jülich Supercomputing Centre , 2015 .

[21]  The International Journal of High Performance Computing Applications— , 1998 .

[22]  J. P. Boris,et al.  New insights into large eddy simulation , 1992 .

[23]  A. Lintermann.  EFFICIENT PARALLEL GEOMETRY DISTRIBUTION FOR THE SIMULATION OF COMPLEX FLOWS , 2016 .

[24]  Klaus Wolf,et al.  MpCCI: Neutral Interfaces for Multiphysics Simulations , 2017, Scientific Computing and Algorithms in Industrial Simulations.

[25]  M. Meinke,et al.  On the accuracy of Lagrangian point-mass models for heavy non-spherical particles in isotropic turbulence , 2017 .

[26]  Wolfgang Schröder,et al.  Effects of tip-gap width on the flow field in an axial fan , 2016 .

[27]  M. Meinke,et al.  Direct particle–fluid simulation of Kolmogorov-length-scale size particles in decaying isotropic turbulence , 2017, Journal of Fluid Mechanics.

[28]  Wolfgang Schröder,et al.  An efficient numerical method for fully‐resolved particle simulations on high‐performance computers , 2015 .

[29]  Thomas Lippert,et al.  The DEEP Project An alternative approach to heterogeneous cluster‐computing in the many‐core era , 2016, Concurr. Comput. Pract. Exp..

[30]  Andreas Lintermann,et al.  A direct-hybrid method for computational aeroacoustics , 2015 .

[31]  C. Moulinec,et al.  Optimizing Code_Saturne computations on Petascale systems , 2011 .

[32]  Florent Duchaine,et al.  Partitioned High Performance Code Coupling Applied to CFD , 2016, JHPCS.

[33]  Wolfgang Schröder,et al.  Cut-cell method based large-eddy simulation of tip-leakage flow , 2015 .

[34]  D. Hartmann,et al.  A level-set based adaptive-grid method for premixed combustion , 2011 .

[35]  Jiri Kraus,et al.  Accelerating a C++ CFD Code with OpenACC , 2014, 2014 First Workshop on Accelerator Programming using Directives.

[36]  M. Meinke,et al.  Lattice-Boltzmann Solutions with Local Grid Refinement for Nasal Cavity Flows , 2013 .

[37]  Wolfgang Schröder,et al.  An accurate moving boundary formulation in cut-cell methods , 2013, J. Comput. Phys..

[38]  Wolfgang Schröder,et al.  Numerical investigation of the three-dimensional flow in a human lung model. , 2008, Journal of biomechanics.

[39]  A. Lintermann,et al.  Comprehensive Visualization of Large-Scale Simulation Data Linked to Respiratory Flow Computations on HPC Systems , 2017 .

[40]  W. Schröder,et al.  Collision rates of small ellipsoids settling in turbulence , 2014, Journal of Fluid Mechanics.

[41]  M. Y. Hussaini,et al.  Discontinuous Spectral Element Approximation of Maxwell’s Equations , 2000 .

[42]  Ernst Hairer,et al.  Solving Ordinary Differential Equations I: Nonstiff Problems , 2009 .

[43]  J. Riley,et al.  Equation of motion for a small rigid sphere in a nonuniform flow , 1983 .

[44]  Mateo Valero,et al.  ALYA: MULTIPHYSICS ENGINEERING SIMULATION TOWARDS EXASCALE , 2014 .

[45]  Andreas Lintermann,et al.  Massively parallel grid generation on HPC systems , 2014 .

[46]  Mike Folk,et al.  Balancing performance and preservation lessons learned with HDF5 , 2010, US-DPIF '10.

[47]  Wolfgang Schröder,et al.  A lattice-Boltzmann method with hierarchically refined meshes , 2013 .

[48]  Andreas Lintermann,et al.  Investigations of Human Nasal Cavity Flows Based on a Lattice-Boltzmann Method , 2011 .

[49]  S. Osher,et al.  A PDE-Based Fast Local Level Set Method 1 , 1998 .

[50]  E. Hairer,et al.  Solving ordinary differential equations I (2nd revised. ed.): nonstiff problems , 1993 .

[51]  Wolfgang Schröder,et al.  An efficient conservative cut-cell method for rigid bodies interacting with viscous compressible flows , 2016, J. Comput. Phys..

[52]  Wolfgang Schröder,et al.  A cut-cell method for sharp moving boundaries in Cartesian grids , 2013 .

[53]  John Shalf,et al.  The International Exascale Software Project roadmap , 2011, Int. J. High Perform. Comput. Appl..

[54]  Y. Qian,et al.  Lattice BGK Models for Navier-Stokes Equation , 1992 .

[55]  Andreas Lintermann,et al.  A Hierarchical Numerical Journey Through the Nasal Cavity: from Nose-Like Models to Real Anatomies , 2019 .

[56]  D. Hartmann,et al.  An adaptive multilevel multigrid formulation for Cartesian hierarchical grid methods , 2008 .

[57]  M. Liou,et al.  A New Flux Splitting Scheme , 1993 .

[58]  Hank Childs,et al.  VisIt: An End-User Tool for Visualizing and Analyzing Very Large Data , 2011 .

[59]  W. Schröder,et al.  Analysis of the Effects of MARME Treatment on Respiratory Flow Using the Lattice-Boltzmann Method , 2018, Notes on Numerical Fluid Mechanics and Multidisciplinary Design.

[60]  P. Lallemand,et al.  Theory of the lattice boltzmann method: dispersion, dissipation, isotropy, galilean invariance, and stability , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[61]  Ronald Fedkiw,et al.  Level set methods and dynamic implicit surfaces , 2002, Applied mathematical sciences.

[62]  Wolfgang Schröder,et al.  Investigation of pulsatile flow in the upper human airways , 2010 .

[63]  Jerry H. Grimmen,et al.  Hydrodynamic instability and shear layer effects in turbulent premixed combustion , 2016 .

[64]  Sabine Roller,et al.  End-to-end Parallel Simulations with APES , 2013, PARCO.

[65]  Matthias Meinke,et al.  A fully coupled hybrid computational aeroacoustics method on hierarchical Cartesian meshes , 2017 .

[66]  Carsten Burstedde,et al.  p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees , 2011, SIAM J. Sci. Comput..

[67]  Florent Duchaine,et al.  Analysis of high performance conjugate heat transfer with the OpenPALM coupler , 2015 .

[68]  Jianwei Li,et al.  Parallel netCDF: A High-Performance Scientific I/O Interface , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[69]  M. Berger,et al.  Progress Towards a Cartesian Cut-Cell Method for Viscous Compressible Flow , 2012 .

[70]  Wolfgang E. Nagel,et al.  Scalable high-quality 1D partitioning , 2014, 2014 International Conference on High Performance Computing & Simulation (HPCS).

[71]  Wolfgang Schröder,et al.  The constrained reinitialization equation for level set methods , 2010, J. Comput. Phys..

[72]  Michael Sturm,et al.  Tonal fan noise of an isolated axial fan rotor due to inhomogeneous coherent structures at the intake , 2012 .

[73]  Wolfgang Schröder,et al.  An Adaptive Cartesian Mesh Based Method to Simulate Turbulent Flows of Multiple Rotating Surfaces , 2017 .

[74]  Jonathan B. Freund,et al.  Proposed Inflow/Outflow Boundary Condition for Direct Computation of Aerodynamic Sound , 1997 .

[75]  D. d'Humières,et al.  Multiple–relaxation–time lattice Boltzmann models in three dimensions , 2002, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[76]  W. Schröder,et al.  Simulation of aerosol particle deposition in the upper human tracheobronchial tract , 2017 .

[77]  S. Osher,et al.  Regular Article: A PDE-Based Fast Local Level Set Method , 1999 .

[78]  P. Lallemand,et al.  Momentum transfer of a Boltzmann-lattice fluid with boundaries , 2001 .

[79]  Dorian Krause,et al.  JURECA: Modular supercomputer at Jülich Supercomputing Centre , 2018, Journal of large-scale research facilities JLSRF.

[80]  Amy Henderson Squilacote.  The Paraview Guide , 2008 .

[81]  M. Meinke,et al.  A flexible level-set approach for tracking multiple interacting interfaces in embedded boundary methods , 2014 .

[82]  Hans-Joachim Bungartz,et al.  preCICE – A fully parallel library for multi-physics surface coupling , 2016 .

[83]  K. D. Beheng,et al.  Numerically determined geometric collision kernels in spatially evolving isotropic turbulence relevant for droplets in clouds , 2013 .

[84]  Wolfgang Schröder,et al.  Erratum to "Differential Equation Based Constrained Reinitialization for Level Set Methods" [J. Comput. Phys. 227(2008) 6821-6845] , 2008, J. Comput. Phys..

[85]  K. Wernecke,et al.  The new agreement of the international RIGA consensus conference on nasal airway function tests. , 2018, Rhinology.

[86]  Derek Gaston,et al.  MOOSE: A parallel computational framework for coupled systems of nonlinear equations , 2009 .

[87]  Wolfgang Schröder,et al.  Analysis of Lattice-Boltzmann methods for internal flows , 2011 .

[88]  J. Bonet,et al.  An alternating digital tree (ADT) algorithm for 3D geometric searching and intersection problems , 1991 .

[89]  P. Bhatnagar,et al.  A Model for Collision Processes in Gases. I. Small Amplitude Processes in Charged and Neutral One-Component Systems , 1954 .

[90]  Wolfgang Joppich,et al.  MpCCI—a tool for the simulation of coupled applications , 2006, Concurr. Comput. Pract. Exp..

[91]  Matthias Meinke,et al.  Efficient parallelization for volume-coupled multiphysics simulations on hierarchical Cartesian grids , 2019, Computer Methods in Applied Mechanics and Engineering.

[92]  James M. Kang,et al.  Space-Filling Curves , 2017, Encyclopedia of GIS.