Higher-order compact scheme for high-performance computing of stratified rotating flows

To take advantage of modern generation computing hardware, a scalable numerical method, based on higher-order compact scheme, is described to solve rotating stratified flows in cylindrical annular domains. An original approach combining 2d-pencil decomposition and reduced Parallel Diagonal Dominant is proposed to improve the parallelization performance during the computation of Poisson/Helmholtz solvers and time explicit terms. The developed technique is validated with respect to analytical solutions, using the method of manufactured solutions, and available data for two specific configurations. The purpose is to demonstrate its ability to correctly capture the flow characteristics in strato-rotational instability and in baroclinic instability with associated small-scale features. Moreover, this code is found to drastically reduce the huge execution times often preventing detailed numerical investigations of these complex phenomena. Strong scaling test is carried out to assess the performance for up to 1024 cores using grid up to 128 × 568 × 568 in radial, axial and azimuthal directions.

[1]  Anthony Randriamampianina,et al.  An improved projection scheme applied to pseudospectral methods for the incompressible Navier-Stokes equations , 1998 .

[2]  Ping He,et al.  A high order finite difference solver for massively parallel simulations of stably stratified turbulent channel flows , 2016 .

[3]  Mateo Valero,et al.  ALYA: MULTIPHYSICS ENGINEERING SIMULATION TOWARDS EXASCALE , 2014 .

[4]  M. Alexander,et al.  Gravity wave dynamics and effects in the middle atmosphere , 2003 .

[5]  Ulrich Achatz,et al.  Gravity wave emission in an atmosphere-like configuration of the differentially heated rotating annulus experiment , 2014, Journal of Fluid Mechanics.

[6]  Spencer J. Sherwin,et al.  Implicit Large-Eddy Simulation of a Wingtip Vortex , 2016 .

[7]  B. Zeghmati,et al.  An efficient parallel high-order compact scheme for the 3D incompressible Navier–Stokes equations , 2017 .

[8]  Vincent Heuveline,et al.  Benchmarking in a rotating annulus: a comparative experimental and numerical study of baroclinic wave dynamics , 2014, 1403.2715.

[9]  J. C. Vassilicos,et al.  A numerical strategy to combine high-order schemes, complex geometry and parallel computing for high resolution DNS of fractal generated turbulence , 2010 .

[10]  Ulrich Achatz,et al.  Spontaneous inertia–gravity wave emission in the differentially heated rotating annulus experiment , 2018, Journal of Fluid Mechanics.

[11]  Rainer Hollerbach,et al.  Magnetic Processes in Astrophysics: Theory, Simulations, Experiments , 2013 .

[12]  Thomas B. Gatski,et al.  A massively parallel hybrid scheme for direct numerical simulation of turbulent viscoelastic channel flow , 2011 .

[13]  M. Gellert,et al.  The stratorotational instability of Taylor-Couette flows with moderate Reynolds numbers , 2016, 1610.02907.

[14]  Y. Notay An aggregation-based algebraic multigrid method , 2010 .

[15]  Stéphane Abide,et al.  A 2D compact fourth-order projection decomposition method , 2005 .

[16]  M. Cox,et al.  Baroclinic Instability in Ocean Currents , 1973 .

[17]  Anthony Randriamampianina,et al.  Instabilities and small-scale waves within the Stewartson layers of a thermally driven rotating annulus , 2018, Journal of Fluid Mechanics.

[18]  Freddie D. Witherden,et al.  PyFR: An open source framework for solving advection-diffusion type problems on streaming architectures using the flux reconstruction approach , 2013, Comput. Phys. Commun..

[19]  Stephen R. Lewis,et al.  Wave interactions and baroclinic chaos: a paradigm for long timescale variability in planetary atmospheres , 1998 .

[20]  P. L. Gal,et al.  Experimental analysis of the stratorotational instability in a cylindrical Couette flow. , 2007, 0707.2661.

[21]  Peter L. Read,et al.  Transition To Geostrophic Turbulence In The Laboratory, And As A Paradigm In Atmospheres And Oceans , 2001 .

[22]  Patrick Bontoux,et al.  A spectral projection method for the simulation of complex three-dimensional rotating flows , 2002 .

[23]  R. Knikker,et al.  Study of a staggered fourth‐order compact scheme for unsteady incompressible viscous flows , 2009 .

[24]  Patrick Knupp,et al.  Code Verification by the Method of Manufactured Solutions , 2000 .

[25]  Robert Michael Kirby,et al.  To CG or to HDG: A Comparative Study in 3D , 2016, J. Sci. Comput..

[26]  Y. Raghu Reddy,et al.  A hybrid MPI-OpenMP scheme for scalable parallel pseudospectral computations for fluid turbulence , 2010, Parallel Comput..

[27]  Roland Schiestel,et al.  A hermitian-fourier numerical method for solving the incompressible navier-stokes equations , 1995 .

[28]  Bilel Hadri,et al.  Scaling of a Fast Fourier Transform and a pseudo-spectral fluid solver up to 196608 cores , 2018, J. Parallel Distributed Comput..

[29]  Todd Gamblin,et al.  Scaling Algebraic Multigrid Solvers: On the Road to Exascale , 2010, CHPC.

[30]  Thomas von Larcher,et al.  Numerical simulations of baroclinic driven flows in a thermally driven rotating annulus using the immersed boundary method , 2015 .

[31]  Belkacem Zeghmati,et al.  A Fourth-Order Iterative Solver for the Singular Poisson Equation , 2012, NAA.

[32]  Liang Shi,et al.  A hybrid MPI-OpenMP parallel implementation for pseudospectral simulations with application to Taylor–Couette flow , 2013, 1311.2481.

[33]  Robert Michael Kirby,et al.  Nektar++: An open-source spectral/hp element framework , 2015, Comput. Phys. Commun..

[34]  Christoph Egbers,et al.  PIV- and LDV-measurements of baroclinic wave interactions in a thermally driven rotating annulus , 2011 .

[35]  Anthony Randriamampianina,et al.  Inertia–gravity waves in a liquid-filled, differentially heated, rotating annulus , 2015, Journal of Fluid Mechanics.

[36]  Ning Li,et al.  2DECOMP&FFT - A Highly Scalable 2D Decomposition Library and FFT Interface , 2010 .

[37]  Dmitry Pekurovsky,et al.  P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions , 2012, SIAM J. Sci. Comput..

[38]  S. Lele Compact finite difference schemes with spectral-like resolution , 1992 .

[39]  Ulrich Rüde,et al.  Massively Parallel Algorithms for the Lattice Boltzmann Method on NonUniform Grids , 2015, SIAM J. Sci. Comput..

[40]  G. Karniadakis,et al.  Spectral/hp Element Methods for Computational Fluid Dynamics , 2005 .

[41]  J. M. Mihaljan,et al.  The baroclinic instability of a , 1962 .

[42]  Michel Deville,et al.  Chebyshev 3-D spectral and 2-D pseudospectral solvers for the Helmholtz equation , 1984 .

[43]  Robert Michael Kirby,et al.  Exploiting Batch Processing on Streaming Architectures to Solve 2D Elliptic Finite Element Problems: A Hybridized Discontinuous Galerkin (HDG) Case Study , 2013, Journal of Scientific Computing.

[44]  Xian-He Sun Application and Accuracy of the Parallel Diagonal Dominant Algorithm , 1995, Parallel Comput..

[45]  M. Gellert,et al.  Stratorotational instability in Taylor–Couette flow heated from above , 2008, Journal of Fluid Mechanics.

[46]  Harry L. Swinney,et al.  Observations of the stratorotational instability in rotating concentric cylinders , 2016, 1604.02963.

[47]  Spencer J. Sherwin,et al.  Optimising the performance of the spectral/hp element method with collective linear algebra operations , 2016 .

[48]  Raymond Hide,et al.  An experimental study of thermal convection in a rotating liquid , 1958, Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences.