Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers

We present an end-to-end optimization of the innovative Arbitrary high-order DERivative Discontinuous Galerkin (ADER-DG) software SeisSol targeting Intel® Xeon Phi coprocessor platforms, achieving unprecedented earthquake model complexity through coupled simulation of full frictional sliding and seismic wave propagation. SeisSol exploits unstructured meshes to flexibly adapt for complicated geometries in realistic geological models. Seismic wave propagation is solved simultaneously with earthquake faulting in a multiphysical manner leading to a heterogeneous solver structure. Our architecture aware optimizations deliver up to 50% of peak performance, and introduce an efficient compute-communication overlapping scheme shadowing the multiphysics computations. SeisSol delivers near-optimal weak scaling, reaching 8.6 DP-PFLOPS on 8,192 nodes of the Tianhe-2 supercomputer. Our performance model projects reaching 18 -- 20 DP-PFLOPS on the full Tianhe-2 machine. Of special relevance to modern civil engineering needs, our pioneering simulation of the 1992 Landers earthquake shows highly detailed rupture evolution and ground motion at frequencies up to 10 Hz.

[1]  Ulrich Rüde,et al.  High performance smart expression template math libraries , 2012, 2012 International Conference on High Performance Computing & Simulation (HPCS).

[2]  David Oglesby,et al.  Short Note Rupture Termination and Jump on Parallel Offset Faults , 2008 .

[3]  Steven M. Day,et al.  Dynamic 3D simulations of earthquakes on En Echelon Faults , 1999 .

[4]  D. Fäh,et al.  Expected seismic shaking in Los Angeles reduced by San Andreas fault zone plasticity , 2014 .

[5]  Thomas H. Heaton,et al.  Near-Source Ground Motions from Simulations of Sustained Intersonic and Supersonic Fault Ruptures , 2004 .

[6]  M. Dumbser,et al.  An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes — II. The three-dimensional isotropic case , 2006 .

[7]  O. Ghattas,et al.  Parallel Octree-Based Finite Element Method for Large-Scale Earthquake Ground Motion Simulation , 2005 .

[8]  Dhabaleswar K. Panda,et al.  MVAPICH-PRISM: A proxy-based communication framework using InfiniBand and SCIF for Intel MIC clusters , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[9]  R. Madariaga,et al.  Dynamic modeling of the 1992 Landers earthquake , 2001 .

[10]  Gregory C. Beroza,et al.  On Scaling of Fracture Energy and Stress Drop in Dynamic Rupture Models: Consequences for Near‐Source Ground‐Motions , 2013 .

[11]  A. Pitarka,et al.  Broadband Ground-Motion Simulation Using a Hybrid Approach , 2010 .

[12]  P. Mai,et al.  Source properties of dynamic rupture pulses with off‐fault plasticity , 2013 .

[13]  Jeremy E. Kozdon,et al.  Rupture to the Trench: Dynamic Rupture Simulations of the 11 March 2011 Tohoku Earthquake , 2013 .

[14]  Vipin Kumar,et al.  A Coarse-Grain Parallel Formulation of Multilevel k-way Graph Partitioning Algorithm , 1997, PP.

[15]  Michael Bader,et al.  Accelerating SeisSol by Generating Vectorized Code for Sparse Matrix Operators , 2013, PARCO.

[16]  J. Ampuero,et al.  Three‐dimensional dynamic rupture simulation with a high‐order discontinuous Galerkin method on unstructured tetrahedral meshes , 2012 .

[17]  Robert A. van de Geijn,et al.  BLIS: A Framework for Rapidly Instantiating BLAS Functionality , 2015, ACM Trans. Math. Softw..

[18]  D. Wald,et al.  Spatial and temporal distribution of slip for the 1992 Landers, California, earthquake , 1994, Bulletin of the Seismological Society of America.

[19]  Alice-Agnes Gabriel,et al.  Sustained Petascale Performance of Seismic Simulations with SeisSol on SuperMUC , 2014, ISC.

[20]  Hideo Aochi,et al.  Three‐dimensional nonplanar simulation of the 1992 Landers earthquake , 2002 .

[21]  Pradeep Dubey,et al.  Design and Implementation of the Linpack Benchmark for Single and Multi-node Systems Based on Intel® Xeon Phi Coprocessor , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[22]  P. Segall,et al.  The co-seismic slip distribution of the Landers earthquake , 1994, Bulletin of the Seismological Society of America.

[23]  Xiaodong Chen,et al.  Near-field ground motion from the Landers earthquake , 1995 .

[24]  Norman H. Sleep,et al.  Site Resonance from Strong Ground Motions at Lucerne, California, during the 1992 Landers Mainshock , 2012 .

[25]  D. Helmberger,et al.  Earthquake ruptures modulated by waves in damaged fault zones , 2014 .

[26]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[27]  C. Kesselman,et al.  CyberShake: A Physics-Based Seismic Hazard Model for Southern California , 2011 .

[28]  Michel Campillo,et al.  Frequency domain inversion of strong motions: Application to the 1992 Landers earthquake , 1995 .

[29]  Bernd Mohr,et al.  The Scalasca performance toolset architecture , 2010, Concurr. Comput. Pract. Exp..

[30]  David D. Pollard,et al.  Mechanics of nonplanar faults at extensional steps with application to the 1992 M 7.3 Landers, California, earthquake , 2013 .

[31]  Dhabaleswar K. Panda,et al.  Scalable Earthquake Simulation on Petascale Supercomputers , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[32]  Martin Käser,et al.  Quantitative accuracy analysis of the discontinuous Galerkin method for seismic wave propagation , 2008 .

[33]  Jean-Paul Ampuero,et al.  Macroscopic Asymmetry of Dynamic Rupture on a Bimaterial Interface With Velocity- Weakening Friction , 2006 .

[34]  Martin Käser,et al.  Dynamic rupture modeling on unstructured meshes using a discontinuous Galerkin method , 2009 .

[35]  J. D. Riera,et al.  The Importance of the Dynamic Source Effects on Strong Ground Motion during the 1999 Chi-Chi, Taiwan, Earthquake: Brief Interpretation of the Damage Distribution on Buildings , 2004 .

[36]  Gregory C. Beroza,et al.  Slip distribution of the 1992 Landers earthquake and its implications for earthquake source mechanics , 1994 .

[37]  D. J. Andrews,et al.  Physical Limits on Ground Motion at Yucca Mountain , 2007 .

[38]  P. Maechling,et al.  Strong shaking in Los Angeles expected from southern San Andreas earthquake , 2006 .

[39]  Jun Zhou,et al.  Physics-based seismic hazard analysis on petascale heterogeneous supercomputers , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[40]  Kim B. Olsen,et al.  Three-Dimensional Dynamic Simulation of the 1992 Landers Earthquake , 1997 .

[41]  Alice-Agnes Gabriel,et al.  Verification of an ADER-DG method for complex dynamic rupture problems , 2013 .

[42]  Robert A. van de Geijn,et al.  High-performance implementation of the level-3 BLAS , 2008, TOMS.

[43]  Pradeep Dubey,et al.  Improving Communication Performance and Scalability of Native Applications on Intel Xeon Phi Coprocessor Clusters , 2014, 2014 IEEE 28th International Parallel and Distributed Processing Symposium.

[44]  Ralph J. Archuleta,et al.  Near‐source ground motion from steady state dynamic rupture pulses , 2005 .