Direct numerical simulation of incompressible flows on parallel Octree grids

We introduce an approach for solving the incompressible Navier-Stokes equations on a forest of Octree grids in a parallel environment. The methodology uses the p4est library of Burstedde et al., SIAM J. Sci. Comput., 33(3) (2011) [15] for the construction and the handling of forests of Octree meshes on massively parallel distributed machines and the framework of Mirzadeh et al., J. Comput. Phys., 322 (2016) [55] for the discretizations on Octree data structures. We introduce relevant additional parallel algorithms and provide performance analyses for individual building bricks and for the full solver. We demonstrate strong scaling for the solver up to 32,768 cores for a problem involving O 6.1⇥ 108 computational cells. We illustrate the dynamic adaptive capabilities of our approach by simulating flows past a stationary sphere, flows due to an oscillatory sphere in a closed box and transport of a passive scalar. Without sacrificing accuracy nor spatial resolution in regions of interest, our approach successfully reduces the number of computational cells to (at most) a few percents of uniform grids with equivalent resolution. We also perform a numerical simulation of the turbulent flow in a superhydrophobic channel with unparalleled wall grid resolution in the streamwise and spanwise directions.

[1]  Frédéric Gibou,et al.  A second order accurate projection method for the incompressible Navier-Stokes equations on non-graded adaptive grids , 2006, J. Comput. Phys..

[2]  Francis X. Giraldo,et al.  Strong scaling for numerical weather prediction at petascale with the atmospheric model NUMA , 2015, Int. J. High Perform. Comput. Appl..

[3]  J. R. Philip Flows satisfying mixed no-slip and no-shear conditions , 1972 .

[4]  J. Sethian,et al.  Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations , 1988 .

[5]  M. Vinokur,et al.  On one-dimensional stretching functions for finite-difference calculations. [computational fluid dynamics] , 1983 .

[6]  Carsten Burstedde,et al.  p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees , 2011, SIAM J. Sci. Comput..

[7]  George Biros,et al.  A Parallel Geometric Multigrid Method for Finite Elements on Octree Meshes , 2010, SIAM J. Sci. Comput..

[8]  Carsten Burstedde,et al.  Low-Cost Parallel Algorithms for 2:1 Octree Balance , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium.

[9]  Jon Louis Bentley,et al.  Quad trees a data structure for retrieval on composite keys , 1974, Acta Informatica.

[10]  Martin Kronbichler,et al.  Algorithms and data structures for massively parallel generic adaptive finite element codes , 2011, ACM Trans. Math. Softw..

[11]  P. Colella,et al.  An Adaptive Level Set Approach for Incompressible Two-Phase Flows , 1997 .

[12]  Hans-Joachim Bungartz,et al.  A Parallel Adaptive Cartesian PDE Solver Using Space-Filling Curves , 2006, Euro-Par.

[13]  S. Popinet Gerris: a tree-based adaptive solver for the incompressible Euler equations in complex geometries , 2003 .

[14]  Haecheon Choi,et al.  An immersed-boundary finite-volume method for simulation of heat transfer in complex geometries , 2004 .

[15]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[16]  Hyungmin Park,et al.  A numerical study of the effects of superhydrophobic surface on skin-friction drag in turbulent channel flow , 2013 .

[17]  Srinivas Aluru,et al.  Parallel domain decomposition and load balancing using space-filling curves , 1997, Proceedings Fourth International Conference on High-Performance Computing.

[18]  Jinhee Jeong,et al.  On the identification of a vortex , 1995, Journal of Fluid Mechanics.

[19]  Frédéric Gibou,et al.  An efficient fluid-solid coupling algorithm for single-phase flows , 2009, J. Comput. Phys..

[20]  Frédéric Gibou,et al.  Geometric integration over irregular domains with application to level-set methods , 2007, J. Comput. Phys..

[21]  Michael J. Aftosmis,et al.  3D applications of a Cartesian grid Euler method , 1995 .

[22]  R. Courant,et al.  On the solution of nonlinear hyperbolic differential equations by finite differences , 1952 .

[23]  Donald Meagher,et al.  Geometric modeling using octree encoding , 1982, Comput. Graph. Image Process..

[24]  M. Berger,et al.  Adaptive mesh refinement for hyperbolic partial differential equations , 1982 .

[25]  S. Osher,et al.  Spatially adaptive techniques for level set methods and incompressible flow , 2006 .

[26]  P. Campbell,et al.  Dynamic Octree Load Balancing Using Space-Filling Curves ∗ , 2003 .

[27]  Santi S. Adavani,et al.  Dendro: Parallel algorithms for multigrid and AMR methods on 2:1 balanced octrees , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[28]  Michael Griebel,et al.  Parallel multigrid in an adaptive PDE solver based on hashing and space-filling curves , 1999, Parallel Comput..

[29]  Frank Losasso,et al.  Simulating water and smoke with an octree data structure , 2004, SIGGRAPH 2004.

[30]  Jung Il Choi,et al.  An immersed boundary method for complex incompressible flows , 2007, J. Comput. Phys..

[31]  Chiang Juay Teo,et al.  Analysis of Stokes flow in microchannels with superhydrophobic surfaces containing a periodic array of micro-grooves , 2008 .

[32]  J. Benek,et al.  A flexible grid embedding technique with application to the Euler equations , 1983 .

[33]  Mary F. Wheeler,et al.  A discontinuous Galerkin method with nonoverlapping domain decomposition for the Stokes and Navier-Stokes problems , 2004, Math. Comput..

[34]  S. Osher,et al.  A level set approach for computing solutions to incompressible two-phase flow , 1994 .

[35]  Alexei M. Khokhlov,et al.  Fully Threaded Tree Algorithms for Adaptive Refinement Fluid Dynamics Simulations , 1997, astro-ph/9701194.

[36]  H. Udaykumar,et al.  Sharp interface Cartesian grid method I: An easily implemented technique for 3D moving boundary computations , 2005 .

[37]  Neil D. Sandham,et al.  Stretched Cartesian grids for solution of the incompressible Navier-Stokes equations , 2000 .

[38]  Ronald Fedkiw,et al.  An adaptive discretization of incompressible flow using a multitude of moving Cartesian grids , 2013, J. Comput. Phys..

[39]  Ilya D. Mishev,et al.  FINITE VOLUME METHODS ON VORONOI MESHES , 1998 .

[40]  Hyun Wook Park,et al.  A Numerical Study of the Effects of Superhydrophobic Surfaces on Skin-Friction Drag Reduction in Wall-Bounded Shear Flows , 2015 .

[41]  Miriam Mehl,et al.  Peano - A Traversal and Storage Scheme for Octree-Like Adaptive Cartesian Multiscale Grids , 2011, SIAM J. Sci. Comput..

[42]  Nancy Wilkins-Diehr,et al.  XSEDE: Accelerating Scientific Discovery , 2014, Computing in Science & Engineering.

[43]  C. Peskin Flow patterns around heart valves: A numerical method , 1972 .

[44]  Georg Stadler,et al.  Scalable adaptive mantle convection simulation on petascale supercomputers , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[45]  William Gropp,et al.  Efficient Management of Parallelism in Object-Oriented Numerical Software Libraries , 1997, SciTools.

[46]  C. Ross Ethier,et al.  A high-order discontinuous Galerkin method for the unsteady incompressible Navier-Stokes equations , 2007, J. Comput. Phys..

[47]  Sophie Papst,et al.  Computational Methods For Fluid Dynamics , 2016 .

[48]  George Em Karniadakis,et al.  A semi-Lagrangian high-order method for Navier-Stokes equations , 2001 .

[49]  Jungwoo Kim,et al.  An immersed-boundary finite-volume method for simulations of flow in complex geometries , 2001 .

[50]  Kyle D. Squires,et al.  LES and DES Investigations of Turbulent Flow over a Sphere at Re = 10,000 , 2003 .

[51]  S. Balachandar,et al.  Direct Numerical Simulation of Flow and Heat Transfer From a Sphere in a Uniform Cross-Flow , 2001 .

[52]  D. D. Zeeuw,et al.  An adaptively refined Cartesian mesh solver for the Euler equations , 1993 .

[53]  John B. Bell,et al.  An Adaptive Mesh Projection Method for Viscous Incompressible Flow , 1997, SIAM J. Sci. Comput..

[54]  V. C. Patel,et al.  Flow past a sphere up to a Reynolds number of 300 , 1999, Journal of Fluid Mechanics.

[55]  R. Verzicco,et al.  Combined Immersed-Boundary Finite-Difference Methods for Three-Dimensional Complex Flow Simulations , 2000 .

[56]  Carsten Burstedde,et al.  Recursive Algorithms for Distributed Forests of Octrees , 2014, SIAM J. Sci. Comput..

[57]  J. Strain Semi-Lagrangian Methods for Level Set Equations , 1999 .

[58]  W. Bangerth,et al.  deal.II—A general-purpose object-oriented finite element library , 2007, TOMS.

[59]  Francis Y. Enomoto,et al.  3D automatic Cartesian grid generation for Euler flows , 1993 .

[60]  Mohammad Mirzadeh,et al.  Parallel level-set methods on adaptive tree-based grids , 2016, J. Comput. Phys..

[61]  M. Lai,et al.  An Immersed Boundary Method with Formal Second-Order Accuracy and Reduced Numerical Viscosity , 2000 .

[62]  Frédéric Gibou,et al.  A stable projection method for the incompressible Navier-Stokes equations on arbitrary geometries and adaptive Quad/Octrees , 2015, J. Comput. Phys..

[63]  Rayhaneh Akhavan,et al.  On the mechanism of turbulent drag reduction with super-hydrophobic surfaces , 2015, Journal of Fluid Mechanics.

[64]  F. Harlow,et al.  Numerical Calculation of Time‐Dependent Viscous Incompressible Flow of Fluid with Free Surface , 1965 .

[65]  A. Chorin Numerical solution of the Navier-Stokes equations , 1968 .

[66]  Jonas Laerte Ansoni,et al.  A finite difference method with meshless interpolation for incompressible flows in non-graded tree-based grids , 2019, J. Comput. Phys..

[67]  Rajat Mittal,et al.  A versatile sharp interface immersed boundary method for incompressible flows with complex boundaries , 2008, J. Comput. Phys..

[68]  James R. Stewart,et al.  A framework approach for developing parallel adaptive multiphysics applications , 2004 .

[69]  David R. O'Hallaron,et al.  Scalable Parallel Octree Meshing for TeraScale Applications , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[70]  Frédéric Gibou,et al.  A second order accurate level set method on non-graded adaptive cartesian grids , 2007, J. Comput. Phys..

[71]  Constantine Bekas,et al.  An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth's mantle , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[72]  Steve L. Karman,et al.  SPLITFLOW - A 3D unstructured Cartesian/prismatic grid CFD code for complex geometries , 1995 .

[73]  T. Lu,et al.  Singularities and treatments of elliptic boundary value problems , 2000 .

[74]  Vipin Kumar,et al.  A Parallel Algorithm for Multilevel Graph Partitioning and Sparse Matrix Ordering , 1998, J. Parallel Distributed Comput..

[75]  Danna Zhou,et al.  d. , 1840, Microbial pathogenesis.

[76]  M. Minion,et al.  Accurate projection methods for the incompressible Navier—Stokes equations , 2001 .

[77]  P. Colella,et al.  Local adaptive mesh refinement for shock hydrodynamics , 1989 .