Simulating the Wenchuan Earthquake with Accurate Surface Topography on Sunway TaihuLight

This paper reports our efforts on performing a 50-m resolution earthquake simulation of the Wenchuan Earthquake (Ms 8.0, China) on Sunway TaihuLight. To accurately capture the surface topography, we adopt a curvilinear grid finite-difference method with a traction image free surface implementation and redesign the algorithm to reduce memory access costs for heterogeneous many-core architectures. We then derive a performance model of our algorithm to guide and drive the further optimization and tuning of various parameters using a genetic algorithm. A data layout transformation is also proposed to improve the direct memory access (DMA) efficiency further. Our efforts improve the simulation efficiency from 0.05% to 7.6%, with a sustained performance of 9.07 Pflops using the entire machine of the Sunway TaihuLight (over 10 million cores), and a large-scale simulation of the Wenchuan earthquake with accurate surface topography and improved coda wave effects.

[1]  Chen Ji,et al.  A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[2]  Alice-Agnes Gabriel,et al.  Extreme Scale Multi-Physics Simulations of the Tsunamigenic 2004 Sumatra Megathrust Earthquake , 2017, SC17: International Conference for High Performance Computing, Networking, Storage and Analysis.

[3]  Robert Jankowski,et al.  Non-linear FEM analysis of earthquake-induced pounding between the main building and the stairway tower of the Olive View Hospital , 2009 .

[4]  Dhabaleswar K. Panda,et al.  Efficient Inter-node MPI Communication Using GPUDirect RDMA for InfiniBand Clusters with NVIDIA GPUs , 2013, 2013 42nd International Conference on Parallel Processing.

[5]  Weiguo Liu,et al.  18.9-Pflops Nonlinear Earthquake Simulation on Sunway TaihuLight: Enabling Depiction of 18-Hz and 8-Meter Scenarios , 2017, SC17: International Conference for High Performance Computing, Networking, Storage and Analysis.

[6]  Jeroen Tromp,et al.  High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[7]  Wei Zhang,et al.  Three-dimensional elastic wave numerical modelling in the presence of surface topography by a collocated-grid finite-difference method on curvilinear grids , 2012 .

[8]  Guang Lin,et al.  On Application of the Weak Galerkin Finite Element Method to a Two-Phase Model for Subsurface Flow , 2016, J. Sci. Comput..

[9]  R. King,et al.  A geological and geophysical context for the Wenchuan earthquake of 12 May 2008, Sichuan, People's Republic of China , 2008 .

[10]  Wei Zhang,et al.  Three-dimensional curved grid finite-difference modelling for non-planar rupture dynamics , 2014 .

[11]  Wei Ge,et al.  The Sunway TaihuLight supercomputer: system and applications , 2016, Science China Information Sciences.

[12]  Matthias Christen,et al.  Patus for convenient high-performance stencils: Evaluation in earthquake simulations , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[13]  Alexander Heinecke,et al.  EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method , 2017, ISC.

[14]  Marco Calò,et al.  Homogenization and implementation of a 3D regional velocity model in Mexico for its application in moment tensor inversion of intermediate-magnitude earthquakes , 2017 .

[15]  Yann Klinger,et al.  Coseismic reverse- and oblique-slip surface faulting generated by the 2008 Mw 7.9 Wenchuan earthquake, China , 2009 .

[16]  Wei Zhang,et al.  Traction image method for irregular free surface boundaries in finite difference seismic wave simulation , 2006 .

[17]  Liu Fan,et al.  Design for auto-tuning PID controller based on genetic algorithms , 2009, 2009 4th IEEE Conference on Industrial Electronics and Applications.

[18]  Guangwen Yang,et al.  swDNN: A Library for Accelerating Deep Learning Applications on Sunway TaihuLight , 2017, 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[19]  Peter Messmer,et al.  Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[20]  Christopher R. Houck,et al.  A Genetic Algorithm for Function Optimization: A Matlab Implementation , 2001 .

[21]  Dhabaleswar K. Panda,et al.  Scalable Earthquake Simulation on Petascale Supercomputers , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[22]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[23]  Jun Zhou,et al.  Physics-based seismic hazard analysis on petascale heterogeneous supercomputers , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[24]  Pher Errol Balde Quinay,et al.  Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[25]  Pradeep Dubey,et al.  Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[26]  Alden H. Wright,et al.  Genetic Algorithms for Real Parameter Optimization , 1990, FOGA.

[27]  Chen Ji,et al.  Stress changes from the 2008 Wenchuan earthquake and increased hazard in the Sichuan basin , 2008, Nature.