Scalability of hybrid programming for a CFD code on the Earth Simulator

The Earth Simulator (ES) is an SMP cluster system. There are two types of parallel programming models available on the ES. One is a flat programming model, in which a parallel program is implemented by MPI interfaces only, both within an SMP node and among nodes. The other is a hybrid programming model, in which a parallel program is written by using thread programming within an SMP node and MPI programming among nodes simultaneously. It is generally known that it is difficult to obtain the same high level of performance using the hybrid programming model as can be achieved with the flat programming model.In this paper, we have evaluated scalability of the code for direct numerical simulation of the Navier-Stokes equations on the ES. The hybrid programming model achieves the sustained performance of 346.9 Gflop/s, while the flat programming model achieves 296.4 Gflop/s with 16 PNs of the ES for a DNS problem size of 2563. For small scale problems, however, the hybrid programming model is not as efficient because of microtasking overhead. It is shown that there is an advantage for the hybrid programming model on the ES for the larger size problems.

[1]  Masahiro Yoshida,et al.  Development and achievement of NAL Numerical Wind Tunnel (NWT) for CFD computations , 1994, Proceedings of Supercomputing '94.

[2]  Shiyi Chen,et al.  High‐resolution turbulent simulations using the Connection Machine‐2 , 1992 .

[3]  Mitsuo Yokokawa,et al.  16.4-Tflops Direct Numerical Simulation of Turbulence by a Fourier Spectral Method on the Earth Simulator , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[4]  Siegfried Benkner,et al.  High-level data mapping for clusters of SMPs , 2001, Proceedings 15th International Parallel and Distributed Processing Symposium. IPDPS 2001.

[5]  Mark Bull,et al.  Development of mixed mode MPI / OpenMP applications , 2001, Sci. Program..

[6]  Franck Cappello,et al.  MPI versus MPI+OpenMP on the IBM SP for the NAS Benchmarks , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[7]  R. Pelz.  The parallel Fourier pseudospectral method , 1991 .

[8]  Mitsuo Yokokawa,et al.  An MPI Benchmark Program Library and Its Application to the Earth Simulator , 2002, ISHPC.

[9]  Y. Kaneda,et al.  Anisotropic velocity correlation spectrum at small scales in a homogeneous turbulent shear flow. , 2002, Physical review letters.

[10]  Rolf Rabenseifner,et al.  Communication Bandwidth of Parallel Programming Models on Hybrid Architectures , 2009, ISHPC.

[11]  Mitsuo Yokokawa,et al.  The Earth Simulator system , 2003 .

[12]  Gerhard Wellein,et al.  Communication and Optimization Aspects of Parallel Programming Models on Hybrid Architectures , 2003, Int. J. High Perform. Comput. Appl..

[13]  Kazuhiko Umezawa,et al.  Hardware technology of the Earth Simulator , 2003 .

[14]  Message P Forum,et al.  MPI: A Message-Passing Interface Standard , 1994 .

[15]  Franck Cappello,et al.  Investigating the performance of two programming models for clusters of SMP PCs , 2000, Proceedings Sixth International Symposium on High-Performance Computer Architecture. HPCA-6 (Cat. No.PR00550).

[16]  T. A. Zang,et al.  Spectral methods for fluid dynamics , 1987 .