SERGHEI (SERGHEI-SWE) v1.0: a performance-portable high-performance parallel-computing shallow-water solver for hydrology and environmental hydraulics

Abstract. The Simulation EnviRonment for Geomorphology, Hydrodynamics, and Ecohydrology in Integrated form (SERGHEI) is a multi-dimensional, multi-domain, and multi-physics model framework for environmental and landscape simulation, designed with an outlook towards Earth system modelling. At the core of SERGHEI's innovation is its performance-portable high-performance parallel-computing (HPC) implementation, built from scratch on the Kokkos portability layer, allowing SERGHEI to be deployed, in a performance-portable fashion, in graphics processing unit (GPU)-based heterogeneous systems. In this work, we explore combinations of MPI and Kokkos using OpenMP and CUDA backends. In this contribution, we introduce the SERGHEI model framework and present with detail its first operational module for solving shallow-water equations (SERGHEI-SWE) and its HPC implementation. This module is designed to be applicable to hydrological and environmental problems including flooding and runoff generation, with an outlook towards Earth system modelling. Its applicability is demonstrated by testing several well-known benchmarks and large-scale problems, for which SERGHEI-SWE achieves excellent results for the different types of shallow-water problems. Finally, SERGHEI-SWE scalability and performance portability is demonstrated and evaluated on several TOP500 HPC systems, with very good scaling in the range of over 20 000 CPUs and up to 256 state-of-the art GPUs.

[1]  G. Kesserwani,et al.  (Multi)wavelet-based Godunov-type simulators of flood inundation: static versus dynamic adaptivity , 2022, Advances in Water Resources.

[2]  G. Blöschl,et al.  An integrated GPU-accelerated modeling framework for high-resolution simulations of rural and urban flash floods , 2022, Environ. Model. Softw..

[3]  L. Cea,et al.  Analysis of two sources of variability of basin outflow hydrographs computed with the 2d shallow water model iber: digital terrain model and unstructured mesh size , 2022, Journal of Hydrology.

[4]  Pierfranco Costabile,et al.  Is HEC-RAS 2D accurate enough for storm-event hazard assessment? Lessons learnt from a benchmarking study based on rain-on-grid modelling , 2021, Journal of Hydrology.

[5]  Seungsoo Lee,et al.  SW2D-GPU: A two-dimensional shallow water model accelerated by GPGPU , 2021, Environ. Model. Softw..

[6]  B. Schmalz,et al.  A Systematic Analysis of the Interaction between Rain-on-Grid-Simulations and Spatial Resolution in 2D Hydrodynamic Modeling , 2021, Water.

[7]  D. Patel,et al.  A review on analysis of flood modelling using different numerical models , 2021, Materials Today: Proceedings.

[8]  Andrea Maranzoni,et al.  Review of Historical Dam-Break Events and Laboratory Tests on Real Topography for the Validation of Numerical Models , 2021, Water.

[9]  Wim Vanderbauwhede,et al.  Making legacy Fortran code type safe through automated program transformation , 2021, J. Supercomput..

[10]  T. T. Dullo,et al.  Assessing climate-change-induced flood risk in the Conasauga River watershed: an application of ensemble hydrodynamic inundation modeling , 2021 .

[11]  Pierfranco Costabile,et al.  A 2D-SWEs framework for efficient catchment-scale simulations: Hydrodynamic scaling properties of river networks and implications for non-uniform grids generation , 2021 .

[12]  Katherine J. Evans,et al.  TRITON: A Multi-GPU open source 2D hydrodynamic flood model , 2021, Environ. Model. Softw..

[13]  S. Popinet,et al.  B-flood 1.0: an open-source Saint-Venant model for flash flood simulation using adaptive refinement , 2021, Geoscientific Model Development.

[14]  Md Bulbul Sharif,et al.  Simulation of Hurricane Harvey flood event through coupled hydrologic‐hydraulic models: Challenges and next steps , 2021, Journal of Flood Risk Management.

[15]  D. Liang,et al.  Innovations towards the next generation of shallow flow models , 2021 .

[16]  Torsten Hoefler,et al.  The digital revolution of Earth-system science , 2021, Nature Computational Science.

[17]  Ilhan Özgen-Xian,et al.  A mass-conservative predictor-corrector solution to the 1D Richards equation with adaptive time control , 2021, Journal of Hydrology.

[18]  Georges Kesserwani,et al.  LISFLOOD-FP 8.0: the new discontinuous Galerkin shallow-water solver for multi-core CPUs and GPUs , 2020, Geoscientific Model Development.

[19]  Janice Lynn Ayog,et al.  Second-order discontinuous Galerkin flood model: Comparison with industry-standard finite volume models , 2020, Journal of Hydrology.

[20]  Adam Mann,et al.  Core Concept: Nascent exascale supercomputers offer promise, present challenges , 2020, Proceedings of the National Academy of Sciences.

[21]  Guangwen Yang,et al.  Translating novel HPC techniques into efficient geoscience solutions , 2020, J. Comput. Sci..

[22]  Yu Tong,et al.  A GPU-based numerical model coupling hydrodynamical and morphological processes , 2020 .

[23]  Georges Kesserwani,et al.  (Multi)wavelets increase both accuracy and efficiency of standard Godunov-type hydrodynamic models: Robust 2D approaches , 2020 .

[24]  P. García-Navarro,et al.  Analysis of the performance of a hybrid CPU/GPU 1D2D coupled model for real flood cases , 2020, Journal of Hydroinformatics.

[25]  P. García-Navarro,et al.  A 2D finite volume simulation tool to enable the assessment of combined hydrological and morphodynamical processes in mountain catchments , 2020, Advances in Water Resources.

[26]  Thomas M. Hines,et al.  Performance Evaluation of a Two-Dimensional Flood Model on Heterogeneous High-Performance Computing Architectures , 2020, PASC.

[27]  Butler W. Lampson,et al.  There’s plenty of room at the Top: What will drive computer performance after Moore’s law? , 2020, Science.

[28]  J. Fernández-Pato,et al.  Performance assessment of 2D Zero-Inertia and Shallow Water models for simulating rainfall-runoff processes , 2020 .

[29]  C. Steefel,et al.  Wavelet-based local mesh refinement for rainfall–runoff simulations , 2020, Journal of Hydroinformatics.

[30]  Godehard Sutmann,et al.  Kokkos implementation of an Ewald Coulomb solver and analysis of performance portability , 2020, J. Parallel Distributed Comput..

[31]  Daniel Caviedes-Voullième,et al.  Multiwavelet-based mesh adaptivity with Discontinuous Galerkin schemes: Exploring 2D shallow water problems , 2020 .

[32]  Carol S. Woodward,et al.  Simulating coupled surface–subsurface flows with ParFlow v3.5.0: capabilities, applications, and ongoing development of an open-source, massively parallel, integrated hydrologic model , 2019, Geoscientific Model Development.

[33]  K. J. Evans,et al.  High-performance computing in water resources hydrodynamics , 2020, Journal of Hydroinformatics.

[34]  F. Aureli,et al.  A GPU-Accelerated Shallow-Water Scheme for Surface Runoff Simulations , 2020 .

[35]  Katherine Yelick,et al.  Exascale applications: skin in the game , 2020, Philosophical Transactions of the Royal Society A.

[36]  Pilar García-Navarro,et al.  2D numerical simulation of unsteady flows for large scale floods prediction in real time , 2019 .

[37]  Katharina Kormann,et al.  Evaluation of performance portability frameworks for the implementation of a particle‐in‐cell code , 2019, Concurr. Comput. Pract. Exp..

[38]  Q. Liang,et al.  A full-scale fluvial flood modelling framework based on a high-performance integrated hydrodynamic modelling system (HiPIMS) , 2019, Advances in Water Resources.

[39]  Park,et al.  2D GPU-Accelerated High Resolution Numerical Scheme for Solving Diffusive Wave Equations , 2019, Water.

[40]  Jennifer K. Ryan,et al.  (Multi)wavelets increase both accuracy and efficiency of standard Godunov-type hydrodynamic models , 2019, Advances in Water Resources.

[41]  Günter Blöschl,et al.  A fast second-order shallow water scheme on two-dimensional structured grids over abrupt topography , 2019, Advances in Water Resources.

[42]  Andrew M. Bradley,et al.  HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model , 2019, Geoscientific Model Development.

[43]  A. Bomers,et al.  The influence of grid shape and grid size on hydraulic river modelling performance , 2019, Environmental Fluid Mechanics.

[44]  Hong Huang,et al.  An urban pluvial flood simulation model based on diffusive wave approximation of shallow water equations , 2019 .

[45]  Bobby Minola Ginting,et al.  Central-upwind scheme for 2D turbulent shallow flows using high-resolution meshes with scalable wall functions , 2019, Computers & Fluids.

[46]  M. Morales-Hernández,et al.  The shallow water equations and their application to realistic cases , 2019, Environmental Fluid Mechanics.

[47]  Ilhan Özgen-Xian,et al.  An improved multislope MUSCL scheme for solving shallow water equations on unstructured grids , 2019, Comput. Math. Appl..

[48]  S. Vater,et al.  A limiter‐based well‐balanced discontinuous Galerkin method for shallow‐water flows with wetting and drying: Triangular grids , 2018, International Journal for Numerical Methods in Fluids.

[49]  P. García-Navarro,et al.  Towards transient experimental water surfaces: A new benchmark dataset for 2D shallow water solvers , 2018, Advances in Water Resources.

[50]  Qiuhua Liang,et al.  Efficient surface water flow simulation on static Cartesian grid with local refinement according to key topographic features , 2018, Computers & Fluids.

[51]  Adrián Navas-Montilla,et al.  2D well-balanced augmented ADER schemes for the Shallow Water Equations with bed elevation and extension to the rotating frame , 2018, J. Comput. Phys..

[52]  L. Cea,et al.  An Accelerated Tool for Flood Modelling Based on Iber , 2018, Water.

[53]  Daniel Caviedes-Voullième,et al.  Cellular Automata and Finite Volume solvers converge for 2D shallow flow modelling for hydrological modelling , 2018, Journal of Hydrology.

[54]  Q. Liang,et al.  A new efficient implicit scheme for discretising the stiff friction terms in the shallow water equations , 2018, Advances in Water Resources.

[55]  L. A. Bearup,et al.  Factors controlling seasonal groundwater and solute flux from snow‐dominated basins , 2018, Hydrological Processes.

[56]  Roger P. Pawlowski,et al.  Toward performance portability of the Albany finite element analysis code using the Kokkos library , 2018, Int. J. High Perform. Comput. Appl..

[57]  Mario Ricchiuto,et al.  r-adaptation for Shallow Water flows: conservation, well balancedness, efficiency , 2018 .

[58]  Stéphane Cordier,et al.  FullSWOF: Full Shallow-Water equations for Overland Flow , 2017, J. Open Source Softw..

[59]  Brijesh Verma,et al.  On-Tree Mango Fruit Size Estimation Using RGB-D Images , 2017, Sensors.

[60]  Wim Vanderbauwhede,et al.  Domain-Specific Acceleration and Auto-Parallelization of Legacy Scientific Code in FORTRAN 77 using Source-to-Source Compilation , 2017, Computers & Fluids.

[61]  Sophie Valcke,et al.  Crossing the chasm: how to develop weather and climate models for next generation computers? , 2017 .

[62]  Hans-Joachim Bungartz,et al.  High performance shallow water kernels for parallel overland flow simulations based on FullSWOF2D , 2017, Comput. Math. Appl..

[63]  Luis Samaniego,et al.  The evolution of process-based hydrologic models: Historical challenges and the collective quest for physical realism. , 2017, Hydrology and earth system sciences.

[64]  Murugesu Sivapalan,et al.  From engineering hydrology to Earth system science: milestones in the transformation of hydrologic science , 2017 .

[65]  Alessandro Dal Palù,et al.  A non-uniform efficient grid type for GPU-parallel Shallow Water Equations models , 2017, Environ. Model. Softw..

[66]  T. Chang,et al.  Modeling rainfall-runoff processes using smoothed particle hydrodynamics with mass-varied particles , 2016 .

[67]  D. Liang,et al.  Shallow water equations with depth-dependent anisotropic porosity for subgrid-scale topography , 2016 .

[68]  Benjamin Dewals,et al.  Discretization of the divergence formulation of the bed slope term in the shallow-water equations and consequences in terms of energy balance , 2016 .

[69]  G. Tsakiris,et al.  A hybrid method for flood simulation in small catchments combining hydrodynamic and hydrological techniques , 2016 .

[70]  David G. Tarboton,et al.  An overview of current applications, challenges, and future trends in distributed process-based models in hydrology , 2016 .

[71]  Pilar García-Navarro,et al.  Rainfall/runoff simulation with 2D full shallow water equations: Sensitivity analysis and calibration of infiltration parameters , 2016 .

[72]  Georges Kesserwani,et al.  Benchmarking a multiresolution discontinuous Galerkin shallow water model: Implications for computational hydraulics , 2015 .

[73]  Hyunuk An,et al.  Analysis of an open source quadtree grid shallow water flow solver for flood simulation , 2015 .

[74]  D. Liang,et al.  Upscaling the shallow water model with a novel roughness formulation , 2015, Environmental Earth Sciences.

[75]  Qiuhua Liang,et al.  Contradiction between the C‐property and mass conservation in adaptive grid based shallow flow models: cause and solution , 2015 .

[76]  T. Schulthess.  Programming revisited , 2015, Nature Physics.

[77]  Qiuhua Liang,et al.  An efficient unstructured MUSCL scheme for solving the 2D shallow water equations , 2015, Environ. Model. Softw..

[78]  Knut-Andreas Lie,et al.  Efficient GPU-Implementation of Adaptive Mesh Refinement for the Shallow-Water Equations , 2014, Journal of Scientific Computing.

[79]  Javier Murillo,et al.  GPU implementation of the 2D shallow water equations for the simulation of rainfall/runoff events , 2015, Environmental Earth Sciences.

[80]  Javier Murillo,et al.  An optimized GPU implementation of a 2D free surface simulation model on unstructured meshes , 2014, Adv. Eng. Softw..

[81]  Daniel Sunderland,et al.  Kokkos: Enabling manycore performance portability through polymorphic memory access patterns , 2014, J. Parallel Distributed Comput..

[82]  Alessandro Dal Palù,et al.  GPU-enhanced Finite Volume Shallow Water solver for fast flood simulations , 2014, Environ. Model. Softw..

[83]  Brett F. Sanders,et al.  Mesh type tradeoffs in 2D hydrodynamic modeling of flooding with a Godunov-based flow solver , 2014 .

[84]  Pilar García-Navarro,et al.  A 2D extension of a Large Time Step explicit scheme (CFL > 1) for unsteady problems with wet/dry boundaries , 2014, J. Comput. Phys..

[85]  Jingming Hou,et al.  A model for overland flow and associated processes within the Hydroinformatics Modelling System , 2014 .

[86]  Luke S. Smith,et al.  Towards a generalised GPU/CPU shallow-flow modelling tool , 2013 .

[87]  Qiuhua Liang,et al.  A stable 2D unstructured shallow flow model for simulations of wetting and drying over rough terrains , 2013 .

[88]  Jingming Hou,et al.  A robust well-balanced model on unstructured grids for shallow water flows with wetting and drying over complex topography , 2013 .

[89]  Feng Zhou,et al.  An adaptive moving finite volume scheme for modeling flood inundation over dry and complex topography , 2013 .

[90]  Fabien Marche,et al.  On the well-balanced numerical discretization of shallow water equations on unstructured meshes , 2013, J. Comput. Phys..

[91]  J. Duan,et al.  Two-dimensional depth-averaged finite volume model for unsteady turbulent flow , 2012 .

[92]  Javier Murillo,et al.  A large time step 1D upwind explicit scheme (CFL > 1): Application to shallow water equations , 2012, J. Comput. Phys..

[93]  Javier Murillo,et al.  Augmented versions of the HLL and HLLC Riemann solvers including source terms in one and two dimensions for shallow flow applications , 2012, J. Comput. Phys..

[94]  Javier Murillo,et al.  Influence of mesh structure on 2D full shallow water equations and SCS Curve Number simulation of rainfall/runoff events , 2012 .

[95]  Yong-Sik Cho,et al.  Numerical study on effects of building groups on dam-break flow in urban areas , 2012 .

[96]  C. Simmons,et al.  HydroGeoSphere: A Fully Integrated, Physically Based Hydrological Model , 2012 .

[97]  Qiuhua Liang,et al.  Dynamically adaptive grid based discontinuous Galerkin shallow water model , 2012 .

[98]  Mustafa S. Altinakar,et al.  Efficient shallow water simulations on GPUs: Implementation, visualization, verification, and validation , 2012 .

[99]  R. Barber,et al.  TELEMAC: An efficient hydrodynamics suite for massively parallel architectures , 2011 .

[100]  Ioana Popescu,et al.  Application of a coastal modelling code in fluvial environments , 2011, Environ. Model. Softw..

[101]  Olivier Delestre,et al.  SWASHES: a compilation of shallow water analytic solutions for hydraulic and environmental studies , 2011, 1110.0288.

[102]  Roger A. Falconer,et al.  Numerical assessment of flood hazard risk to people and vehicles in flash floods , 2011, Environ. Model. Softw..

[103]  David L. George,et al.  Adaptive finite volume methods with well‐balanced Riemann solvers for modeling floods in rugged terrain: Application to the Malpasset dam‐break flood (France, 1959) , 2011 .

[104]  Jim W Hall,et al.  A 2D shallow flow model for practical dam-break simulations , 2011 .

[105]  Norbert Silvera,et al.  Comparison of roughness models to simulate overland flow and tracer transport experiments under simulated rainfall at plot scale , 2011 .

[106]  Qiuhua Liang,et al.  Well-balanced RKDG2 solutions to the shallow water equations over irregular domains with wetting and drying , 2010 .

[107]  J Puertas,et al.  Overland flow computations in urban and industrial catchments from direct precipitation data using a two-dimensional shallow water model. , 2010, Water science and technology : a journal of the International Association on Water Pollution Research.

[108]  Randall J. LeVeque,et al.  The GeoClaw software for depth-averaged flows with adaptive refinement , 2010, 1008.0455.

[109]  Javier Murillo,et al.  Weak solutions for partial differential equations with source terms: Application to the shallow water equations , 2010, J. Comput. Phys..

[110]  Jerónimo Puertas,et al.  Experimental validation of two-dimensional depth-averaged models for forecasting rainfall-runoff from precipitation data in urban areas , 2010 .

[111]  I. Nikolos,et al.  An unstructured node-centered finite volume scheme for shallow water flows with wet/dry fronts over complex topography , 2009 .

[112]  André Paquier,et al.  Modelling flash flood propagation in urban areas using a two-dimensional numerical model , 2009 .

[113]  Javier Murillo,et al.  Time step restrictions for well‐balanced shallow water solutions in non‐zero velocity steady states , 2009 .

[114]  Javier Murillo,et al.  A finite volume method for the simulation of the waves generated by landslides , 2009 .

[115]  Javier Murillo,et al.  Friction term discretization and limitation to preserve stability and conservation in the 1D shallow‐water model: Application to unsteady irrigation and river flow , 2008 .

[116]  Yves Zech,et al.  Dam-break flow through an idealised city , 2008 .

[117]  F. Aureli,et al.  A weighted surface-depth gradient method for the numerical integration of the 2D shallow water equations with topography , 2008 .

[118]  Sandra Soares-Frazão,et al.  Experiments of dam-break wave over a triangular bottom sill , 2007 .

[119]  Efim Pelinovsky,et al.  Three‐dimensional simulation of tsunami run‐up around conical island , 2007 .

[120]  T. A. Black,et al.  Net ecosystem productivity of boreal jack pine stands regenerating from clearcutting under current and future climates , 2007 .

[121]  Azzeddine Soulaïmani,et al.  Numerical Tracking of Shallow Water Waves by the Unstructured Finite Volume WAF Approximation , 2007 .

[122]  Roger Alexander Falconer,et al.  A boundary-fitted numerical model for flood routing with shock capturing , 2007 .

[123]  U. C. Kothyari,et al.  A GIS based distributed rainfall–runoff model , 2004 .

[124]  Pilar García-Navarro,et al.  Zero mass error using unsteady wetting–drying conditions in shallow flows over dry irregular topography , 2004 .

[125]  J. Hervouet,et al.  Malpasset dam-break revisited with two-dimensional computations , 1999 .

[126]  P. Roe.  Approximate Riemann Solvers, Parameter Vectors, and Difference Schemes , 1997 .

[127]  Alessandro Dal Palù,et al.  A local time stepping algorithm for GPU-accelerated 2D shallow water models , 2018 .

[128]  V. G. Jetten,et al.  The validity of flow approximations when simulating catchment-integrated flash floods , 2018 .

[129]  J. Banfield,et al.  The East River, Colorado, Watershed: A Mountainous Community Testbed for Improving Predictive Understanding of Multiscale Hydrological–Biogeochemical Dynamics , 2018 .

[130]  K. Ando,et al.  Parallel computing for high-resolution/large-scale flood simulation using the K supercomputer , 2015 .

[131]  Satish Karra,et al.  PFLOTRAN User Manual A Massively Parallel Reactive Flow and Transport Model for Describing Surface and Subsurface Processes , 2015 .

[132]  G. Gottardi,et al.  An accurate time integration method for simplified overland flow models , 2008 .

[133]  John Wainwright,et al.  Measurement and modelling of high resolution flow-velocity data under simulated rainfall on a low-slope sandy soil , 2008 .