Parallel DNS algorithms on unstructured grids

With the prospect of Petaflop computing to be realized in the near future, we present two different parallel algorithms suitable for simulating turbulent flows in non-separable and multiply-connected computational domains. The algorithms are based on a new class of hierarchical spectral methods appropriate for tensor-product representations in hybrid subdomains, i.e., tetrahedra, hexahedra, prisms and pyramids. We review the numerical implementation of the spectral method and subsequently present two parallel paradigms, the first for a spectral element/Fourier algorithm, and the second for a fully 3D algorithm based on geometric domain decomposition. Emphasis is placed on the communication patterns of these algorithms in conjunction with the features of current or upcoming computer models. Examples of turbulent parallel simulations are included and limitations in currently achieving high parallel efficiencies are discussed. A perspective on the future of DNS on the emerging distributed shared memory (DSM) computer architectures is presented.

[1]  John S. Keen,et al.  Measuring Memory Hierarchy Performance of Cache-Coherent Multiprocessors Using Micro Benchmarks , 1997, ACM/IEEE SC 1997 Conference (SC'97).

[2]  George Em Karniadakis,et al.  Parallel benchmarks of turbulence in complex geometries , 1996 .

[3]  Rupak Biswas,et al.  Tetrahedral and hexahedral mesh adaptation for CFD problems , 1998 .

[4]  Spencer J. Sherwin,et al.  Hierarchical hp finite elements in hybrid domains , 1997 .

[5]  George Karypis,et al.  Multilevel k-way Partitioning Scheme for Irregular Graphs , 1998, J. Parallel Distributed Comput..

[6]  George E. Karniadakis,et al.  Parallel Cfd Benchmarks on Cray Computers , 1996, Parallel Algorithms Appl..

[7]  P. Moin,et al.  Direct numerical simulation of transition and turbulence in a spatially evolving boundary layer , 1991 .

[8]  John B. McLaughlin,et al.  Large‐scale computer simulation of fully developed turbulent channel flow with heat transfer , 1991 .

[9]  Barry F. Smith,et al.  Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations , 1996 .

[10]  P. Moin,et al.  Turbulence statistics in fully developed channel flow at low Reynolds number , 1987, Journal of Fluid Mechanics.

[11]  George Em Karniadakis,et al.  A direct numerical simulation of laminar and turbulent flow over riblet-mounted surfaces , 1993, Journal of Fluid Mechanics.

[12]  George Em Karniadakis,et al.  Unstructured hp/Spectral Elements: Connectivity and Optimal Ordering , 1995 .

[13]  Spencer J. Sherwin,et al.  Helix And Model Graft Flows: Mri Measurement And Cfd Simulations , 1997 .

[14]  George Em Karniadakis,et al.  A triangular spectral element method; applications to the incompressible Navier-Stokes equations , 1995 .

[15]  R. S. Reichert,et al.  Applications of direct numerical simulation to complex turbulent flows , 1997 .

[16]  L. Dagum,et al.  OpenMP: an industry standard API for shared-memory programming , 1998 .

[17]  S. Parekh,et al.  Tuning Compiler Optimizations for Simultaneous Multithreading , 1997, Proceedings of 30th Annual International Symposium on Microarchitecture.

[18]  Steven G. Johnson,et al.  The Fastest Fourier Transform in the West (MIT-LCS-TR-728) , 1997 .

[19]  I. Babuska,et al.  Finite Element Analysis , 2021 .

[20]  Anshul Gupta,et al.  Fast and effective algorithms for graph partitioning and sparse-matrix ordering , 1997, IBM J. Res. Dev..

[21]  Martin G. Everett,et al.  Parallel Dynamic Graph Partitioning for Adaptive Unstructured Meshes , 1997, J. Parallel Distributed Comput..

[22]  Fong Pong,et al.  Missing the Memory Wall: The Case for Processor/Memory Integration , 1996, 23rd Annual International Symposium on Computer Architecture (ISCA'96).

[23]  R.A. Fiedler.  Optimization and Scaling of Shared-Memory and Message-Passing Implementations of the Zeus Hydrodynamics Algorithm , 1997, ACM/IEEE SC 1997 Conference (SC'97).

[24]  George Em Karniadakis,et al.  A NEW TRIANGULAR AND TETRAHEDRAL BASIS FOR HIGH-ORDER (HP) FINITE ELEMENT METHODS , 1995 .

[25]  Simon Kahan,et al.  Tera hardware-software cooperation , 1997, SC '97.

[26]  Beth Ann Wingate.  Triangular spectral elements for geophysical fluid dynamics applications. , 1996 .

[27]  Jack J. Dongarra,et al.  Message-Passing Performance of Various Computers , 1997, Concurr. Pract. Exp..

[28]  James J. Riley,et al.  Direct numerical simulation of turbulent reacting flow using a reduced hydrogenoxygen mechanism , 1993 .

[29]  S. Orszag,et al.  High-order splitting methods for the incompressible Navier-Stokes equations , 1991 .

[30]  George Em Karniadakis,et al.  Unstructured spectral element methods for simulation of turbulent flows , 1995 .

[31]  Paul Fischer,et al.  Algorithms for large-scale parallel simulation of unsteady incompressible flows in three-dimensional complex geometries , 1998 .

[32]  George Em Karniadakis,et al.  TetrahedralhpFinite Elements , 1996 .

[33]  Dean M. Tullsen,et al.  Simultaneous multithreading: a platform for next-generation processors , 1997, IEEE Micro.

[34]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[35]  P. Moin,et al.  Numerical experiments on the flow past A circular cylinder at sub-critical reynolds number , 1994 .

[36]  Moshe Dubiner.  Spectral methods on triangles and other domains , 1991 .

[37]  Chris Walshaw,et al.  Load-balancing for parallel adaptive unstructured grids , 1998 .

[38]  Bruce Hendrickson,et al.  An Improved Spectral Graph Partitioning Algorithm for Mapping Parallel Computations , 1995, SIAM J. Sci. Comput..

[39]  Wayne Smith,et al.  Multigrid solution of the Euler equations , 1987 .

[40]  Leonid Oliker,et al.  Efficient load balancing and data remapping for adaptive grid calculations , 1997, SPAA '97.

[41]  Steven L. Scott,et al.  Synchronization and communication in the T3E multiprocessor , 1996, ASPLOS VII.

[42]  Jack Dongarra,et al.  Message-Passing Performance of Various Computers , 1995 .

[43]  Timothy C. Warburton,et al.  Basis Functions for Triangular and Quadrilateral High-Order Elements , 1999, SIAM J. Sci. Comput..

[44]  George Em Karniadakis,et al.  Nodes, modes and flow codes , 1993 .

[45]  Leszek Demkowicz,et al.  Toward a universal h-p adaptive finite element strategy , 1989 .

[46]  P. Fischer,et al.  PARALLEL SIMULATION OF VISCOUS INCOMPRESSIBLE FLOWS , 1994 .

[47]  Gheith A. Abandah,et al.  Modeling the communication performance of the IBM SP2 , 1996, Proceedings of International Conference on Parallel Processing.

[48]  Martin G. Everett,et al.  Parallel dynamic load-balancing for adaptive unstructured meshes , 1997, Parallel CFD.

[49]  S. Sherwin,et al.  The influence of out-of-plane geometry on the flow within a distal end-to-side anastomosis. , 2000, Journal of biomechanical engineering.

[50]  G. S. Patterson,et al.  Numerical simulation of turbulence , 1972 .