152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquake simulation

We have used data learning and low-precision computation to develop an implicit solver that demonstrates high performance up to 152,352 computer nodes (609,408 MPI processes × 12 OpenMP threads = 7,312,896 parallel computation) and conducted an unprecedented ultra-large-scale analysis of ultra-high-fidelity fault-structure systems using nonlinear dynamic finite element analysis on three-dimensional low-order unstructured elements. The developed solver achieved 25.45-fold speedup from the state of the art solver on Fugaku and attained weak scaling efficiency of 93.7% from 9.391 billion DOF@578 computer nodes to 1.201 trillion DOF@73,984 computer nodes on performance measurement problems. Moreover, a realistic 324 billion DOF application example, which is difficult to obtain performance for, was computed in high performance. Since the developed solver is based on a highly generalizable algorithm, it is expected to contribute not only to earthquake simulation on Fugaku but also to the enhancement of similar applications in other fields and on other supercomputers.

[1]  Eric Darve,et al.  Assembly of finite element methods on graphics processors , 2011 .

[2]  Peng Wang,et al.  High-Frequency Nonlinear Earthquake Simulations on Petascale Heterogeneous Supercomputers , 2016, SC16: International Conference for High Performance Computing, Networking, Storage and Analysis.

[3]  Uri Shaham,et al.  Dynamic Mode Decomposition , 2013 .

[4]  Tsuyoshi Ichimura,et al.  Physics-Based Urban Earthquake Simulation Enhanced by 10.7 BlnDOF × 30 K Time-Step Unstructured FE Non-Linear Seismic Wave Simulation , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[5]  Thomas J. R. Hughes,et al.  Solution algorithms for nonlinear transient heat conduction analysis employing element-by-element iterative strategies , 1985 .

[6]  Y. Kodama,et al.  Co-Design for A64FX Manycore Processor and ”Fugaku” , 2020, SC20: International Conference for High Performance Computing, Networking, Storage and Analysis.

[7]  Pher Errol Balde Quinay,et al.  Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[8]  Jacobo Bielak,et al.  Coupled Soil-Structure Interaction Effects of Building Clusters during Earthquakes , 2015 .

[9]  Peter Messmer,et al.  Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[10]  Pher Errol Balde Quinay,et al.  Comprehensive numerical analysis of fault‐structure systems – Computation of the large‐scale seismic structural response to a given earthquake scenario – , 2012 .

[11]  Takayuki Okamoto,et al.  The Tofu Interconnect D , 2018, 2018 IEEE International Conference on Cluster Computing (CLUSTER).

[12]  Doriam Restrepo,et al.  A computational workflow for rupture‐to‐structural‐response simulation and its application to Istanbul , 2020, Earthquake Engineering & Structural Dynamics.

[13]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[14]  Izzat M. Idriss,et al.  NONLINEAR BEHAVIOR OF SOFT CLAYS DURING CYCLIC LOADING , 1978 .

[15]  Jun Zhou,et al.  Physics-based seismic hazard analysis on petascale heterogeneous supercomputers , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[16]  Steven L. Brunton,et al.  Dynamic mode decomposition - data-driven modeling of complex systems , 2016 .

[17]  Tjerk P. Straatsma,et al.  A Fast Scalable Implicit Solver for Nonlinear Time-Evolution Earthquake City Problem on Low-Ordered Unstructured Finite Elements with Artificial Intelligence and Transprecision Computing , 2018, SC18: International Conference for High Performance Computing, Networking, Storage and Analysis.

[18]  Nathan M. Newmark,et al.  A Method of Computation for Structural Dynamics , 1959 .

[19]  G. Masing,et al.  Eigenspannungen und Verfestigung beim Messing , 1926 .

[20]  Gerhard J. Woeginger,et al.  Graph colorings , 2005, Theor. Comput. Sci..

[21]  Gene H. Golub,et al.  Inexact Preconditioned Conjugate Gradient Method with Inner-Outer Iteration , 1999, SIAM J. Sci. Comput..

[22]  Pradeep Dubey,et al.  Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[23]  Alice-Agnes Gabriel,et al.  Extreme Scale Multi-Physics Simulations of the Tsunamigenic 2004 Sumatra Megathrust Earthquake , 2017, SC17: International Conference for High Performance Computing, Networking, Storage and Analysis.

[24]  Thomas H. Jordan.  Societal Impact of Earthquake Simulations at Extreme Scale , 2015, HiPC 2015.

[25]  Tsuyoshi Ichimura,et al.  Wave Propagation Simulation of Complex Multi-Material Problems with Fast Low-Order Unstructured Finite-Element Meshing and Analysis , 2018, HPC Asia.

[26]  Tsuyoshi Ichimura,et al.  A hybrid multiresolution meshing technique for finite element three‐dimensional earthquake ground motion modelling in basins including topography , 2009 .

[27]  Tsuyoshi Ichimura,et al.  High-fidelity nonlinear low-order unstructured implicit finite-element seismic simulation of important structures by accelerated element-by-element method , 2021, J. Comput. Sci..