A Simulation of Seismic Wave Propagation at High Resolution in the Inner Core of the Earth on 2166 Processors of MareNostrum

We use 2166 processors of the MareNostrum (IBM PowerPC 970) supercomputer to model seismic wave propagation in the inner core of the Earth following an earthquake. Simulations are performed based upon the spectral-element method, a high-degree finite-element technique with an exactly diagonal mass matrix. We use a mesh with 21 billion grid points (and therefore approximately 21 billion degrees of freedom because a scalar unknown is used in most of the mesh). A total of 2.5 terabytes of memory is needed. Our implementation is purely based upon MPI. We optimize it using the ParaVer analysis tool in order to significantly improve load balancing and therefore overall performance. Cache misses are reduced based upon renumbering of the mesh points.

[1]  D. Komatitsch,et al.  Spectral-element simulations of global seismic wave propagation—I. Validation , 2002 .

[2]  R. Sadourny.  Conservative Finite-Difference Approximations of the Primitive Equations on Quasi-Uniform Spherical Grids , 1972 .

[3]  D. Komatitsch,et al.  The spectral element method: An efficient tool to simulate the seismic response of 2D and 3D geological structures , 1998, Bulletin of the Seismological Society of America.

[4]  Karl E. Butler,et al.  Seismoelectric imaging of the vadose zone of a sand aquifer , 2007 .

[5]  Chen Ji,et al.  A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[6]  Jesús Labarta,et al.  Performance analysis of multilevel parallel applications on shared memory architectures , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[7]  Géza Seriani,et al.  3-D large-scale wave propagation modeling by spectral element method on Cray T3E multiprocessor , 1998 .

[8]  E. Cuthill,et al.  Reducing the bandwidth of sparse symmetric matrices , 1969, ACM '69.

[9]  Jean-Pierre Vilotte,et al.  Solving elastodynamics in a fluid-solid heterogeneous sphere: a parallel spectral element approximation on non-conforming grids , 2003 .

[10]  Mrinal K. Sen,et al.  Grid dispersion and stability criteria of some common finite-element methods for acoustic and elastic wave equations , 2007 .

[11]  D. Komatitsch,et al.  Spectral-element simulations of global seismic wave propagation: II. Three-dimensional models, oceans, rotation and self-gravitation , 2002 .

[12]  D. Komatitsch,et al.  Introduction to the spectral element method for three-dimensional seismic wave propagation , 1999 .

[13]  Kent T. Danielson,et al.  Nonlinear dynamic finite element analysis on parallel computers using FORTRAN 90 and MPI 1 This pape , 1998 .