10M-Core Scalable Fully-Implicit Solver for Nonhydrostatic Atmospheric Dynamics

An ultra-scalable fully-implicit solver is developed for stiff time-dependent problems arising from the hyperbolic conservation laws in nonhydrostatic atmospheric dynamics. In the solver, we propose a highly efficient hybrid domain-decomposed multigrid preconditioner that can greatly accelerate the convergence rate at the extreme scale. For solving the overlapped subdomain problems, a geometry-based pipelined incomplete LU factorization method is designed to further exploit the on-chip fine-grained concurrency. We perform systematic optimizations on different hardware levels to achieve best utilization of the heterogeneous computing units and substantial reduction of data movement cost. The fully-implicit solver successfully scales to the entire system of the Sunway TaihuLight supercomputer with over 10.5M heterogeneous cores, sustaining an aggregate performance of 7.95 PFLOPS in double-precision, and enables fast and accurate atmospheric simulations at the 488-m horizontal resolution (over 770 billion unknowns) with 0.07 simulated-years-per-day. This is, to our knowledge, the largest fully-implicit simulation to date.

[1]  Clive Temperton,et al.  An Efficient Two‐Time‐Level Semi‐Lagrangian Semi‐Implicit Integration Scheme , 1987 .

[2]  Constantine Bekas,et al.  An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth's mantle , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[3]  Edmond Chow,et al.  Fine-Grained Parallel Incomplete LU Factorization , 2015, SIAM J. Sci. Comput..

[4]  Peter Bauer,et al.  The quiet revolution of numerical weather prediction , 2015, Nature.

[5]  Adrian Sandu,et al.  Multi-core acceleration of chemical kinetics for simulation and prediction , 2009, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis.

[6]  Christiane Jablonowski,et al.  Operator-Split Runge-Kutta-Rosenbrock Methods for Nonhydrostatic Atmospheric Models , 2012 .

[7]  Hirofumi Tomita,et al.  Performance Analysis and Optimization of Nonhydrostatic ICosahedral Atmospheric Model (NICAM) on the K Computer and TSUBAME2.5 , 2016, PASC.

[8]  Mark A. Taylor,et al.  CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model , 2012, Int. J. High Perform. Comput. Appl..

[9]  Xiao-Chuan Cai,et al.  A Restricted Additive Schwarz Preconditioner for General Sparse Linear Systems , 1999, SIAM J. Sci. Comput..

[10]  Olof B. Widlund,et al.  The Development of Coarse Spaces for Domain Decomposition Algorithms , 2009 .

[11]  Chao Yang,et al.  A Scalable Fully Implicit Compressible Euler Solver for Mesoscale Nonhydrostatic Simulation of Atmospheric Flows , 2014, SIAM J. Sci. Comput..

[12]  Chao Yang,et al.  A peta-scalable CPU-GPU algorithm for global atmospheric simulations , 2013, PPoPP '13.

[13]  J. Klemp,et al.  The Simulation of Three-Dimensional Convective Storm Dynamics , 1978 .

[14]  YangChao,et al.  A peta-scalable CPU-GPU algorithm for global atmospheric simulations , 2013 .

[15]  N. Phillips,et al.  Scale Analysis of Deep and Shallow Convection in the Atmosphere , 1962 .

[16]  Rory Kelly,et al.  GPU Computing for Atmospheric Modeling , 2010, Computing in Science & Engineering.

[17]  Richard W. Vuduc,et al.  Petascale Direct Numerical Simulation of Blood Flow on 200K Cores and Heterogeneous Architectures , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[18]  Division on Earth.  A National Strategy for Advancing Climate Modeling , 2013 .

[19]  André Robert,et al.  A SEMI-IMPLICIT SCHEME FOR GRID POINT ATMOSPHERIC MODELS OF THE PRIMITIVE EQUATIONS , 1971 .

[20]  Chao Yang,et al.  Ultra-Scalable CPU-MIC Acceleration of Mesoscale Atmospheric Modeling on Tianhe-2 , 2015, IEEE Transactions on Computers.

[21]  Alan Norton,et al.  Petascale WRF simulation of hurricane sandy: Deployment of NCSA's cray XE6 blue waters , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[22]  Dana A. Knoll,et al.  An Implicit Nonlinearly Consistent Method for the Two-Dimensional Shallow-Water Equations with Coriolis Force , 2002 .

[23]  William Gropp,et al.  High-performance parallel implicit CFD , 2001, Parallel Comput..

[24]  Hiroaki Miura,et al.  Global cloud‐system‐resolving model NICAM successfully simulated the lifecycles of two real tropical cyclones , 2008 .

[25]  Masaki Satoh,et al.  Nonhydrostatic icosahedral atmospheric model (NICAM) for global cloud resolving simulations , 2008, J. Comput. Phys..

[26]  Edmond Chow,et al.  Iterative Sparse Triangular Solves for Preconditioning , 2015, Euro-Par.

[27]  Chao Yang,et al.  Parallel multilevel methods for implicit solution of shallow water equations with nonsmooth topography on the cubed-sphere , 2011, J. Comput. Phys..

[28]  Masaki Satoh,et al.  Conservative scheme for the compressible nonhydrostatic models with the horizontally explicit and vertically implicit time integration scheme , 2002 .

[29]  Omar M. Knio,et al.  Regime of Validity of Soundproof Atmospheric Flow Models , 2010 .

[30]  Chao Yang,et al.  Enabling and Scaling a Global Shallow-Water Atmospheric Model on Tianhe-2 , 2014, 2014 IEEE 28th International Parallel and Distributed Processing Symposium.

[31]  P. Lauritzen.  Numerical techniques for global atmospheric models , 2011 .

[32]  Takashi Shimokawabe,et al.  145 TFlops Performance on 3990 GPUs of TSUBAME 2.0 Supercomputer for an Operational Weather Prediction , 2011, ICCS.

[33]  Pher Errol Balde Quinay,et al.  Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[34]  Wei Ge,et al.  The Sunway TaihuLight supercomputer: system and applications , 2016, Science China Information Sciences.

[35]  D. Waldman,et al.  U.S. Economic Sensitivity to Weather Variability , 2011 .