Fourier spectral/hp element method: investigation of time-stepping and parallelisation strategies

As computer hardware has evolved, the time required to perform numerical simulations has reduced, allowing investigations of a wide range of new problems. This thesis focuses on algorithm optimisation, to minimise run-time, when solving the incompressible Navier-Stokes equations. Aspects affecting performance related to the discretisation and algorithm parallelisation are investigated in the context of high-order methods. The roles played by numerical approximations and computational strategies are highlighted and it is recognised that a versatile implementation provides additional benefits, allowing an ad-hoc selection of techniques to fit the needs of heterogeneous computing environments. We initially describe the building blocks of a spectral/hp element and pure spectral method and how they can be encapsulated and combined to create a 3D discretisation, the Fourier spectral/hp element method. Time-stepping strategies are also described and encapsulated in a flexible framework based on the General Linear Method. After implementing and validating an incompressible Navier-Stokes solver, two canonical turbulent flows are analysed. Afterward a 2D hyperbolic equation is considered to investigate the efficiency of lowand high-order methods when discretising the spatial and temporal derivatives. We perform parametric studies, monitoring accuracy and CPU-time for different numerical approximations. We identify optimal discretisations, demonstrating that high-order methods are the computationally fastest approach to attain a desired accuracy for this problem. Following the same philosophy, we investigate the benefits of using a hybrid parallel implementation. The message passing model is introduced to parallelise different kernels of an incompressible Navier-Stokes solver. Monitoring the parallel performance of these strategies the most efficient approach is highlighted. We also demonstrate that hybrid parallel solutions can be used to significantly extend the strong scalability limit and support greater parallelism.

[1]  Mark Ainsworth,et al.  Dispersive and Dissipative Properties of Discontinuous Galerkin Finite Element Methods for the Second-Order Wave Equation , 2006, J. Sci. Comput..

[2]  Jason Sanders,et al.  CUDA by example: an introduction to general purpose GPU programming , 2010 .

[3]  Rajeev Thakur,et al.  Communication analysis of parallel 3D FFT for flat cartesian meshes on large Blue Gene systems , 2008, HiPC'08.

[4]  Spencer J. Sherwin,et al.  Formulation of a Galerkin spectral element-fourier method for three-dimensional incompressible flows in cylindrical geometries , 2004 .

[5]  J. Hesthaven,et al.  Nodal Discontinuous Galerkin Methods: Algorithms, Analysis, and Applications , 2007 .

[6]  Chunlei Liang,et al.  A comparison of computational efficiencies of spectral difference method and correction procedure via reconstruction , 2013, J. Comput. Phys..

[7]  Einar M. Rønquist,et al.  Optimal spectral element methods for the unsteady three-dimensional incompressible Navier-Stokes equations , 1988 .

[8]  S. Sherwin,et al.  Direct optimal growth analysis for timesteppers , 2008 .

[9]  George Em Karniadakis,et al.  Parallel benchmarks of turbulence in complex geometries , 1996 .

[10]  Spencer J. Sherwin,et al.  Flow-induced vibration of a circular cylinder subjected to wake interference at low Reynolds number , 2011 .

[11]  John Kim,et al.  DIRECT NUMERICAL SIMULATION OF TURBULENT CHANNEL FLOWS UP TO RE=590 , 1999 .

[12]  DEALIASED CONVOLUTIONS FOR PSEUDOSPECTRAL SIMULATIONS , 2011 .

[13]  P. Fischer,et al.  PARALLEL SIMULATION OF VISCOUS INCOMPRESSIBLE FLOWS , 1994 .

[14]  Barbara Chapman,et al.  Using OpenMP: Portable Shared Memory Parallel Programming (Scientific and Engineering Computation) , 2007 .

[15]  Chi-Wang Shu,et al.  The Local Discontinuous Galerkin Method for Time-Dependent Convection-Diffusion Systems , 1998 .

[16]  Mark Ainsworth,et al.  Discrete Dispersion Relation for hp-Version Finite Element Approximation at High Wave Number , 2004, SIAM J. Numer. Anal..

[17]  M Israeli,et al.  Numerical Simulation of Viscous Incompressible Flows , 1974 .

[18]  Jie Shen,et al.  An overview of projection methods for incompressible flows , 2006 .

[19]  John Kim,et al.  On the structure of pressure fluctuations in simulated turbulent channel flow , 1989, Journal of Fluid Mechanics.

[20]  Chi-Wang Shu,et al.  Strong Stability-Preserving High-Order Time Discretization Methods , 2001, SIAM Rev..

[21]  G. S. Patterson,et al.  Spectral Calculations of Isotropic Turbulence: Efficient Removal of Aliasing Interactions , 1971 .

[22]  Spencer J. Sherwin,et al.  A generic framework for time-stepping partial differential equations (PDEs): general linear methods, object-oriented implementation and application to fluid problems , 2011 .

[23]  Martin Berzins,et al.  Parallelization and scalability of a spectral element channel flow solver for incompressible Navier–Stokes equations , 2007, Concurr. Comput. Pract. Exp..

[24]  T. A. Zang,et al.  Spectral Methods: Fundamentals in Single Domains , 2010 .

[25]  Hans Werner Meuer,et al.  Top500 Supercomputer Sites , 1997 .

[26]  Spencer J. Sherwin,et al.  On discontinuous Galerkin methods , 2003 .

[27]  E. Toro,et al.  An arbitrary high-order Discontinuous Galerkin method for elastic waves on unstructured meshes - V. Local time stepping and p-adaptivity , 2007 .

[28]  S. Sherwin,et al.  Transient growth mechanisms of low Reynolds number flow over a low-pressure turbine blade , 2011, 1302.2770.

[29]  den Jmj Jaap Toonder,et al.  Reynolds number effects in a turbulent pipe flow for low to moderate Re , 1997 .

[30]  Robert Michael Kirby,et al.  From h to p efficiently: Implementing finite and spectral/hp element methods to achieve optimal performance for low- and high-order discretisations , 2010, J. Comput. Phys..

[31]  Ning Li,et al.  2DECOMP&FFT - A Highly Scalable 2D Decomposition Library and FFT Interface , 2010 .

[32]  George Em Karniadakis,et al.  Galerkin and discontinuous Galerkin spectral/hp methods , 1999 .

[33]  P. Moin,et al.  Numerical investigation of turbulent channel flow , 1981, Journal of Fluid Mechanics.

[34]  C. Canuto.  Spectral methods in fluid dynamics , 1991 .

[35]  Peter Vos.  From h to p efficiently : optimising the implementation of spectral/hp element methods , 2011 .

[36]  S. Orszag,et al.  High-order splitting methods for the incompressible Navier-Stokes equations , 1991 .

[37]  Timothy C. Warburton,et al.  Taming the CFL Number for Discontinuous Galerkin Methods on Structured Meshes , 2008, SIAM J. Numer. Anal..

[38]  David A. Ham,et al.  Finite element assembly strategies on multi‐core and many‐core architectures , 2013 .

[39]  Paul F. Fischer,et al.  Fast Parallel Direct Solvers for Coarse Grid Problems , 2001, J. Parallel Distributed Comput..

[40]  Gregory A. Blaisdell,et al.  The effect of the formulation of nonlinear terms on aliasing errors in spectral methods , 1996 .

[41]  C. Hirsch,et al.  Numerical Computation of Internal and External Flows. By C. HIRSCH. Wiley. Vol. 1, Fundamentals of Numerical Discretization. 1988. 515 pp. £60. Vol. 2, Computational Methods for Inviscid and Viscous Flows. 1990, 691 pp. £65. , 1991, Journal of Fluid Mechanics.

[42]  S. Sherwin,et al.  Pulse wave propagation in a model human arterial network: Assessment of 1-D visco-elastic simulations against in vitro measurements , 2011, Journal of biomechanics.

[43]  Mrinal K. Sen,et al.  Stability of the high-order finite elements for acoustic or elastic wave propagation with high-order time stepping , 2010 .

[44]  Wolfgang Henrik T. Koberg.  Turbulence control for drag reduction with active wall deformation , 2008 .

[45]  S. Orszag.  Spectral methods for problems in complex geometries , 1980 .

[46]  Torsten Hoefler,et al.  MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared memory , 2013, Computing.

[47]  Mrinal K. Sen,et al.  The interior penalty discontinuous Galerkin method for elastic wave propagation: grid dispersion , 2008 .

[48]  W. L. Wood.  Practical Time-Stepping Schemes , 1990 .

[49]  Richard D. Sandberg,et al.  Efficient parallel computing with a compact finite difference scheme , 2011 .

[50]  Vipin Kumar,et al.  The Scalability of FFT on Parallel Computers , 1993, IEEE Trans. Parallel Distributed Syst..

[51]  L. Collatz.  The numerical treatment of differential equations , 1961 .

[52]  S. Sherwin,et al.  From h to p efficiently: optimal implementation strategies for explicit time-dependent problems using the spectral/hp element method , 2014, International journal for numerical methods in fluids.

[53]  Y. Raghu Reddy,et al.  A hybrid MPI-OpenMP scheme for scalable parallel pseudospectral computations for fluid turbulence , 2010, Parallel Comput..

[54]  Claudio Canuto,et al.  Spectral Methods: Evolution to Complex Geometries and Applications to Fluid Dynamics (Scientific Computation) , 2007 .

[55]  Paul Fischer,et al.  Analysis and application of a parallel spectral element method for the solution of the Navier-Stokes equations , 1990 .

[56]  Dror G. Feitelson.  On the Interpretation of Top500 Data , 1999, Int. J. High Perform. Comput. Appl..

[57]  S. Sherwin,et al.  From h to p efficiently: Strategy selection for operator evaluation on hexahedral and tetrahedral elements , 2011 .

[58]  R. Rogallo.  Numerical experiments in homogeneous turbulence , 1981 .

[59]  Claus-Dieter Munz,et al.  An explicit discontinuous Galerkin scheme with local time-stepping for general unsteady diffusion equations , 2008, J. Comput. Phys..

[60]  Antony Jameson,et al.  On the Development of a High-Order, Multi-GPU Enabled, Compressible Viscous Flow Solver for Mixed Unstructured Grids , 2011 .

[61]  Antony Jameson,et al.  Energy stable flux reconstruction schemes for advection-diffusion problems on triangles , 2013, J. Comput. Phys..

[62]  George Em Karniadakis,et al.  A triangular spectral element method; applications to the incompressible Navier-Stokes equations , 1995 .

[63]  Steven G. Johnson,et al.  The Design and Implementation of FFTW3 , 2005, Proceedings of the IEEE.

[64]  George E. Karniadakis,et al.  Communication Performance Models in Prism : A Spectral Element-Fourier Parallel Navier-Stokes Solver , 1996, Proceedings of the 1996 ACM/IEEE Conference on Supercomputing.

[65]  S. Sherwin,et al.  STABILISATION OF SPECTRAL/HP ELEMENT METHODS THROUGH SPECTRAL VANISHING VISCOSITY: APPLICATION TO FLUID MECHANICS MODELLING , 2006 .

[66]  Paul Fischer,et al.  Spectral element methods for large scale parallel Navier—Stokes calculations , 1994 .

[67]  John C. Bowman,et al.  Efficient Dealiased Convolutions without Padding , 2011, SIAM J. Sci. Comput..

[68]  A. Quarteroni,et al.  Non-conforming high order approximations of the elastodynamics equation , 2012 .

[69]  J. Butcher.  The numerical analysis of ordinary differential equations: Runge-Kutta and general linear methods , 1987 .

[70]  G. Karniadakis,et al.  Spectral/hp Element Methods for Computational Fluid Dynamics , 2005 .

[71]  Robert Michael Kirby,et al.  Parallel Scientific Computing in C++ and MPI - A Seamless Approach to Parallel Algorithms and their Implementation , 2003 .

[72]  Mark Ainsworth,et al.  Dispersive and dissipative behaviour of high order discontinuous Galerkin finite element methods , 2004 .

[73]  Harold L. Atkins,et al.  Eigensolution analysis of the discontinuous Galerkin method with non-uniform grids , 2001 .

[74]  A. Patera.  A spectral element method for fluid dynamics: Laminar flow in a channel expansion , 1984 .

[75]  G. Karniadakis.  Spectral element—Fourier methods for incompressible turbulent flows , 1990 .

[76]  Jesús Carretero,et al.  Design and implementation of a parallel I/O runtime system for irregular applications , 1998, Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing.

[77]  Marco Lúcio Bittencourt,et al.  SPECTRAL/HP FINITE ELEMENTS APPLIED TO LINEAR AND NON-LINEAR STRUCTURAL ELASTIC PROBLEMS , 2007 .

[78]  Steven J. Ruuth,et al.  Implicit-explicit methods for time-dependent partial differential equations , 1995 .

[79]  D. Gottlieb,et al.  Numerical analysis of spectral methods : theory and applications , 1977 .

[80]  W. H. Reed,et al.  Triangular mesh methods for the neutron transport equation , 1973 .

[81]  Vipin Kumar,et al.  Isoefficiency: measuring the scalability of parallel algorithms and architectures , 1993, IEEE Parallel & Distributed Technology: Systems & Applications.

[82]  Chi-Wang Shu,et al.  Runge–Kutta Discontinuous Galerkin Methods for Convection-Dominated Problems , 2001, J. Sci. Comput..

[83]  Spencer J. Sherwin,et al.  From h to p Efficiently: Selecting the Optimal Spectral/hp Discretisation in Three Dimensions , 2011 .

[84]  Spencer J. Sherwin,et al.  Dispersion Analysis of the Continuous and Discontinuous Galerkin Formulations , 2000 .

[85]  M. Dumbser,et al.  An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes — II. The three-dimensional isotropic case , 2006 .

[86]  P. Fischer,et al.  Petascale algorithms for reactor hydrodynamics , 2008 .

[87]  John C. Butcher,et al.  General linear methods for ordinary differential equations , 2009, Math. Comput. Simul..

[88]  Spectral element-Fourier methods applied to simulation of turbulent pipe flow , 2000 .

[89]  Torsten Hoefler,et al.  Hybrid MPI: Efficient message passing for multi-core systems , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[90]  Daisuke Takahashi.  Efficient implementation of parallel three-dimensional FFT on clusters of PCs , 2003 .

[91]  George Bosilca,et al.  Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation , 2004, PVM/MPI.

[92]  John C. Butcher.  General linear methods , 2006, Acta Numerica.

[93]  Todd E. Peterson,et al.  A note on the convergence of the discontinuous Galerkin method for a scalar hyperbolic equation , 1991 .

[94]  P. Moin,et al.  Turbulence statistics in fully developed channel flow at low Reynolds number , 1987, Journal of Fluid Mechanics.

[95]  Paul Fischer,et al.  An Overlapping Schwarz Method for Spectral Element Solution of the Incompressible Navier-Stokes Equations , 1997 .

[96]  Chi-Wang Shu,et al.  Stability Analysis and A Priori Error Estimates of the Third Order Explicit Runge-Kutta Discontinuous Galerkin Method for Scalar Conservation Laws , 2010, SIAM J. Numer. Anal..

[97]  Ewing L. Lusk,et al.  Early Experiments with the OpenMP/MPI Hybrid Programming Model , 2008, IWOMP.

[98]  Bengt Fornberg,et al.  A practical guide to pseudospectral methods: Introduction , 1996 .

[99]  James Demmel,et al.  Parallel numerical linear algebra , 1993, Acta Numerica.