18.9-Pflops Nonlinear Earthquake Simulation on Sunway TaihuLight: Enabling Depiction of 18-Hz and 8-Meter Scenarios

This paper reports our large-scale nonlinear earthquake simulation software on Sunway TaihuLight. Our innovations include: (1) a customized parallelization scheme that employs the 10 million cores efficiently at both the process and the thread levels; (2) an elaborate memory scheme that integrates on-chip halo exchange through register communcation, optimized blocking configuration guided by an analytic model, and coalesced DMA access with array fusion; (3) on-the-fly compression that doubles the maximum problem size and further improves the performance by 24%. With these innovations to remove the memory constraints of Sunway TaihuLight, our software achieves over 15% of the system's peak, better than the 11.8% efficiency achieved by a similar software running on Titan, whose byte to flop ratio is 5 times better than TaihuLight. The extreme cases demonstrate a sustained performance of over 18.9 Pflops, enabling the simulation of Tangshan earthquake as an 18-Hz scenario with an 8-meter resolution.

[1]  John Milne,et al.  Earthquakes and other Earth Movements , 1913, Nature.

[2]  Chen Ji,et al.  A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[3]  Pradeep Dubey,et al.  3.5-D Blocking Optimization for Stencil Computations on Modern CPUs and GPUs , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[4]  Jeroen Tromp,et al.  High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[5]  David R. O'Hallaron,et al.  Earthquake ground motion modeling on parallel computers , 1996, Supercomputing '96.

[6]  Qiu Ze-hua.  DISCOVERY OF THE GREAT FAULT OF THE TANGSHAN EARTHQUAKE , 2005 .

[7]  Jing Zhu,et al.  Enabling Very-Large Scale Earthquake Simulations on Parallel Machines , 2007, International Conference on Computational Science.

[8]  Peng Wang,et al.  High-Frequency Nonlinear Earthquake Simulations on Petascale Heterogeneous Supercomputers , 2016, SC16: International Conference for High Performance Computing, Networking, Storage and Analysis.

[9]  Pher Errol Balde Quinay,et al.  Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[10]  Pradeep Dubey,et al.  Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[11]  Wei Ge,et al.  The Sunway TaihuLight supercomputer: system and applications , 2016, Science China Information Sciences.

[12]  Alexander Heinecke,et al.  EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method , 2017, ISC.

[13]  Wei Zhang,et al.  Three-dimensional curved grid finite-difference modelling for non-planar rupture dynamics , 2014 .

[14]  Xie Xin-sheng.  NEW EVIDENCE FOR THE DISTRIBUTION OF SURFACE RUPTURE ZONE OF THE 1976 M_S 7.8 TANGSHAN EARTHQUAKE , 2011 .

[15]  Hong-Sen Yan,et al.  THE REVIEW OF RECONSTRUCTION DESIGNS OF ZHANG HENG'S SEISMOSCOPE , 2009 .

[16]  Tsuyoshi Ichimura,et al.  Physics-Based Urban Earthquake Simulation Enhanced by 10.7 BlnDOF × 30 K Time-Step Unstructured FE Non-Linear Seismic Wave Simulation , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[17]  Qi-yuan Liu,et al.  Seismogenic Tectonic Environment of 1976 Great Tangshan Earthquake: Results from Dense Seismic Array Observations , 2007 .

[18]  Charles E. Catlett.  The Philosophy of TeraGrid: Building an Open, Extensible, Distributed TeraScale Facility , 2002, CCGRID.

[19]  Peter Messmer,et al.  Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[20]  Dhabaleswar K. Panda,et al.  Scalable Earthquake Simulation on Petascale Supercomputers , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[21]  M. Wysession,et al.  An Introduction to Seismology, Earthquakes, and Earth Structure , 2002 .

[22]  Jun Zhou,et al.  Physics-based seismic hazard analysis on petascale heterogeneous supercomputers , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).