A Low-Communication-Overhead Parallel DNS Method for the 3D Incompressible Wall Turbulence

This paper presents a low-communication-overhead parallel method for direct numerical simulations of the 3D incompressible wall turbulence. A fully explicit projection method with second-order space-time accuracy is adopted. Combined with fast Fourier transforms, the parallel diagonal dominant (PDD) algorithm for the tridiagonal system is employed to solve the pressure Poisson equation. The number of all-to-all communications is decreased to only two, in a 2D pencil-like domain decomposition. The resulting MPI/OpenMP hybrid parallel code shows excellent strong scalability up to 104 cores and small wall-clock time per timestep. Numerical simulations of turbulent channel flow at different friction Reynolds numbers ( ) have been conducted and the statistics are in good agreement with the reference data and the recent scaling theories [Chen, X. and K. R. Sreenivasan. 2021. “Reynolds Number Scaling of the Peak Turbulence Intensity in Wall Flows.” Journal of Fluid Mechanics 908: R3. doi:10.1017/jfm.2020.991].

[1]  Donghyun You,et al.  Application of the parallel diagonal dominant algorithm for the incompressible Navier-Stokes equations , 2020, J. Comput. Phys..

[2]  J. Jiménez,et al.  Effect of the computational domain on direct simulations of turbulent channels up to Reτ = 4200 , 2014 .

[3]  Truong Vinh Truong Duy,et al.  A decomposition method with minimum communication amount for parallelization of multi-dimensional FFTs , 2014, Comput. Phys. Commun..

[4]  R. Lahey,et al.  Direct numerical simulation of turbulent channel flows using a stabilized finite element method , 2009 .

[5]  Min Xie,et al.  Auto-Tuning MPI Collective Operations on Large-Scale Parallel Systems , 2019, 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS).

[6]  Z. She,et al.  Non-universal scaling transition of momentum cascade in wall turbulence , 2019, Journal of Fluid Mechanics.

[7]  R. Sweet,et al.  Fast Fourier transforms for direct solution of Poisson's equation with staggered boundary conditions , 1988 .

[8]  P. Moin,et al.  Turbulence statistics in fully developed channel flow at low Reynolds number , 1987, Journal of Fluid Mechanics.

[9]  O. Lehmkuhl,et al.  Noise radiated by an open cavity at low Mach number: Effect of the cavity oscillation mode , 2019, International Journal of Aeroacoustics.

[10]  P. Moin,et al.  The minimal flow unit in near-wall turbulence , 1991, Journal of Fluid Mechanics.

[11]  J. Jiménez,et al.  Hierarchy of minimal flow units in the logarithmic layer , 2010 .

[12]  Javier Jiménez,et al.  Scaling of the velocity fluctuations in turbulent channels up to Reτ=2003 , 2006 .

[13]  B. Zeghmati,et al.  An efficient parallel high-order compact scheme for the 3D incompressible Navier–Stokes equations , 2017 .

[14]  Endong Wang,et al.  Intel Math Kernel Library , 2014 .

[15]  J. Luo,et al.  Parallel Direct Method of DNS for Two-Dimensional Turbulent Rayleigh-Bénard Convection , 2018 .

[16]  Hari Sundar,et al.  FFT, FMM, or Multigrid? A comparative Study of State-Of-the-Art Poisson Solvers for Uniform and Nonuniform Grids in the Unit Cube , 2014, SIAM J. Sci. Comput..

[17]  John Kim,et al.  DIRECT NUMERICAL SIMULATION OF TURBULENT CHANNEL FLOWS UP TO RE=590 , 1999 .

[18]  Roger W. Hockney,et al.  A Fast Direct Solution of Poisson's Equation Using Fourier Analysis , 1965, JACM.

[19]  Gianluca Iaccarino,et al.  IMMERSED BOUNDARY METHODS , 2005 .

[20]  R. Moser,et al.  Direct numerical simulation of turbulent channel flow up to $\mathit{Re}_{{\it\tau}}\approx 5200$ , 2014, Journal of Fluid Mechanics.

[21]  Pedro Costa,et al.  A FFT-based finite-difference solver for massively-parallel direct numerical simulations of turbulent flows , 2018, Comput. Math. Appl..

[22]  Roberto Verzicco,et al.  A pencil distributed finite difference code for strongly turbulent wall-bounded flows , 2015, 1501.01247.

[23]  Steven G. Johnson,et al.  The Design and Implementation of FFTW3 , 2005, Proceedings of the IEEE.

[24]  P. Orlandi,et al.  Velocity statistics in turbulent channel flow up to $Re_{\tau }=4000$ , 2014, Journal of Fluid Mechanics.

[25]  Jeffrey K. Hollingsworth,et al.  Computation-communication overlap and parameter auto-tuning for scalable parallel 3-D FFT , 2016, J. Comput. Sci..

[26]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[27]  R. Verzicco,et al.  Combined Immersed-Boundary Finite-Difference Methods for Three-Dimensional Complex Flow Simulations , 2000 .

[28]  K. Sreenivasan,et al.  Reynolds number scaling of the peak turbulence intensity in wall flows , 2020, Journal of Fluid Mechanics.

[29]  F. Hussain,et al.  Drag control in wall-bounded turbulent flows via spanwise opposed wall-jet forcing , 2018, Journal of Fluid Mechanics.

[30]  R. Narasimha,et al.  New insights from high-resolution compressible DNS studies on an LPT blade boundary layer , 2017 .

[31]  Massimiliano Fatica,et al.  AFiD-GPU: A versatile Navier-Stokes solver for wall-bounded turbulent flows on GPU clusters , 2017, Comput. Phys. Commun..

[32]  Javier Jiménez,et al.  Cascades in Wall-Bounded Turbulence , 2012 .

[33]  Carlos David Pérez Segarra,et al.  Direct Numerical Simulation of the flow over a sphere at Re = 3700 , 2009 .

[34]  Lionel M. Ni,et al.  Parallel algorithms for solution of tridiagonal systems on multicomputers , 1989, ICS '89.

[35]  Martin Kronbichler,et al.  A high-order semi-explicit discontinuous Galerkin solver for 3D incompressible flow with application to DNS and LES of turbulent channel flow , 2016, J. Comput. Phys..

[36]  B. Cantwell.  A universal velocity profile for smooth wall pipe flow , 2019, Journal of Fluid Mechanics.

[37]  A. W. Vreman,et al.  Comparison of direct numerical simulation databases of turbulent channel flow at Re τ = 180 , 2014 .

[38]  Harold S. Stone,et al.  An Efficient Parallel Algorithm for the Solution of a Tridiagonal Linear System of Equations , 1973, JACM.

[39]  H. H. Wang,et al.  A Parallel Method for Tridiagonal Equations , 1981, TOMS.

[40]  Yoshinobu Yamamoto,et al.  Numerical evidence of logarithmic regions in channel flow at Reτ=8000 , 2018 .

[41]  Michael Pippig.  PFFT: An Extension of FFTW to Massively Parallel Architectures , 2013, SIAM J. Sci. Comput..

[42]  Detlef Lohse,et al.  The near-wall region of highly turbulent Taylor–Couette flow , 2015, Journal of Fluid Mechanics.

[43]  Xian-He Sun.  Application and Accuracy of the Parallel Diagonal Dominant Algorithm , 1995, Parallel Comput..

[44]  Z. She,et al.  Quantifying wall turbulence via a symmetry approach. Part 2. Reynolds stresses , 2018, Journal of Fluid Mechanics.

[45]  Ning Li,et al.  2DECOMP&FFT - A Highly Scalable 2D Decomposition Library and FFT Interface , 2010 .

[46]  A. Smits,et al.  Wall-bounded turbulent flows at high Reynolds numbers: Recent advances and key issues , 2010 .

[47]  Duncan H. Lawrie,et al.  The computation and communication complexity of a parallel banded system solver , 1984, TOMS.

[48]  Dmitry Pekurovsky,et al.  P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions , 2012, SIAM J. Sci. Comput..