Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x)

Abstract. One of the fundamental factors contributing to the spatiotemporal inaccuracy in climate modeling is the mapping of solution field data between different discretizations and numerical grids used in the coupled component models. The typical climate computational workflow involves evaluation and serialization of the remapping weights during the preprocessing step, which is then consumed by the coupled driver infrastructure during simulation to compute field projections. Tools like Earth System Modeling Framework (ESMF) ( Hill et al. ,  2004 ) and TempestRemap ( Ullrich et al. ,  2013 ) offer capability to generate conservative remapping weights, while the Model Coupling Toolkit (MCT) ( Larson et al. ,  2001 ) that is utilized in many production climate models exposes functionality to make use of the operators to solve the coupled problem. However, such multistep processes present several hurdles in terms of the scientific workflow and impede research productivity. In order to overcome these limitations, we present a fully integrated infrastructure based on the Mesh Oriented datABase (MOAB) ( Tautges et al. ,  2004 ; Mahadevan et al. ,  2015 ) library, which allows for a complete description of the numerical grids and solution data used in each submodel. Through a scalable advancing-front intersection algorithm, the supermesh of the source and target grids are computed, which is then used to assemble the high-order, conservative, and monotonicity-preserving remapping weights between discretization specifications. The Fortran-compatible interfaces in MOAB are utilized to directly link the submodels in the Energy Exascale Earth System Model (E3SM) to enable online remapping strategies in order to simplify the coupled workflow process. We demonstrate the superior computational efficiency of the remapping algorithms in comparison with other state-of-the-science tools and present strong scaling results on large-scale machines for computing remapping weights between the spectral element atmosphere and finite volume discretizations on the polygonal ocean grids.

[1]  Malcolm Roberts,et al.  Developing the next-generation climate system models: challenges and achievements , 2009, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[2]  Hester Bijl,et al.  Comparison of the conservative and a consistent approach for the coupling of non-matching meshes , 2006 .

[3]  T. Reichler,et al.  How Well Do Coupled Models Simulate Today's Climate? , 2008 .

[4]  Cecelia DeLuca,et al.  The architecture of the Earth System Modeling Framework , 2003, Computing in Science & Engineering.

[5]  Stephen J. Thomas,et al.  A mass and energy conserving spectral element atmospheric dynamical core on the cubed-sphere grid , 2007 .

[6]  Philip W. Jones,et al.  A multi-resolution approach to global ocean modeling , 2013 .

[7]  Timothy J. Tautges,et al.  Scalable parallel solution coupling for multiphysics reactor simulation , 2009 .

[8]  Moritz Hanke,et al.  YAC 1.2.0: new aspects for coupling software in Earth system modelling , 2016 .

[9]  Jan Brezina,et al.  Fast algorithms for intersection of non-matching grids using Plücker coordinates , 2017, Comput. Math. Appl..

[10]  Michael T. Heath,et al.  Common‐refinement‐based data transfer between non‐matching meshes in multiphysics simulations , 2004 .

[11]  Bengt Fornberg,et al.  On choosing a radial basis function and a shape parameter when solving a convective PDE on a sphere , 2008, J. Comput. Phys..

[12]  Mariana Vertenstein,et al.  A new flexible coupler for earth system modeling developed for CCSM4 and CESM1 , 2012, Int. J. High Perform. Comput. Appl..

[13]  Stuart R. Slattery,et al.  Mesh-free data transfer algorithms for partitioned multiphysics problems: Conservation, accuracy, and parallelism , 2016, J. Comput. Phys..

[14]  Günther Zängl,et al.  The ICON-1.2 hydrostatic atmospheric dynamical core on triangular grids – Part 1: Formulation and performance of the baseline version , 2013 .

[15]  Stephen J. Thomas,et al.  The NCAR Spectral Element Climate Dynamical Core: Semi-Implicit Eulerian Formulation , 2005, J. Sci. Comput..

[16]  Philip W. Jones.  First- and Second-Order Conservative Remapping Schemes for Grids in Spherical Coordinates , 1999 .

[17]  Christiane Jablonowski,et al.  Geometrically Exact Conservative Remapping (GECoRe): Regular Latitude–Longitude and Cubed-Sphere Grids , 2009 .

[18]  Chi-Wang Shu,et al.  On the Gibbs Phenomenon and Its Resolution , 1997, SIAM Rev..

[19]  Spencer Rugaber,et al.  A feature model of coupling technologies for Earth System Models , 2013, Comput. Geosci..

[20]  Michael Garland,et al.  Implementing sparse matrix-vector multiplication on throughput-oriented processors , 2009, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis.

[21]  Courtenay T. Vaughan,et al.  Zoltan data management services for parallel dynamic applications , 2002, Comput. Sci. Eng..

[22]  Tom Abel,et al.  An exact general remeshing scheme applied to physically conservative voxelization , 2014, J. Comput. Phys..

[23]  M. Berger.  ON CONSERVATION AT GRID INTERFACES. , 1987 .

[24]  Paresh Parikh,et al.  Generation of three-dimensional unstructured grids by the advancing-front method , 1988 .

[25]  Todd D. Ringler,et al.  Evaluation of the arbitrary Lagrangian–Eulerian vertical coordinate method in the MPAS-Ocean model , 2015 .

[26]  Bin Wang,et al.  C-Coupler1: a Chinese community coupler for Earth system modeling , 2014 .

[27]  Guangwen Yang,et al.  C-Coupler2: a flexible and user-friendly community coupler for model coupling and nesting , 2018 .

[28]  Chris H. Q. Ding,et al.  CPL6: The New Extensible, High Performance Parallel Coupler for the Community Climate System Model , 2005, Int. J. High Perform. Comput. Appl..

[29]  Christiane Jablonowski,et al.  Some considerations for high-order 'incremental remap'-based transport schemes: Edges, reconstructions, and area integration , 2013 .

[30]  S. Valcke,et al.  The OASIS3 coupler: a European climate modelling community software , 2012 .

[31]  Steven J. Plimpton,et al.  A Parallel Rendezvous Algorithm for Interpolation Between Multiple Grids , 1998, Proceedings of the IEEE/ACM SC98 Conference.

[32]  W.R. Mark,et al.  Fast kd-tree Construction with an Adaptive Error-Bounded Heuristic , 2006, 2006 IEEE Symposium on Interactive Ray Tracing.

[33]  Erwin Laure,et al.  Performance Analysis of Irregular Collective Communication with the Crystal Router Algorithm , 2014, EASC.

[34]  Natasha Flyer,et al.  Transport schemes on a sphere using radial basis functions , 2007, J. Comput. Phys..

[35]  Jeffrey Grandy.  Conservative Remapping and Region Overlays by Intersecting Arbitrary Polyhedra , 1999 .

[36]  Sophie Valcke,et al.  Development and performance of a new version of the OASIS coupler, OASIS3-MCT_3.0 , 2017 .

[37]  Mark A. Taylor,et al.  CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model , 2012, Int. J. High Perform. Comput. Appl..

[38]  Dharshi Devendran,et al.  Arbitrary-Order Conservative and Consistent Remapping and a Theory of Linear Maps: Part II , 2015 .

[39]  Paul A. Ullrich,et al.  A conservative semi-Lagrangian multi-tracer transport scheme (CSLAM) on the cubed-sphere grid , 2010, J. Comput. Phys..

[40]  Dmitriy Morozov,et al.  Block-parallel data analysis with DIY2 , 2016, 2016 IEEE 6th Symposium on Large Data Analysis and Visualization (LDAV).

[41]  Rainald Löhner,et al.  Recent Advances in Parallel Advancing Front Grid Generation , 2014, Archives of Computational Methods in Engineering.

[42]  S. J. Zhou.  Coupling climate models with the Earth System Modeling Framework and the Common Component Architecture , 2006 .

[43]  D. Cohen-Or,et al.  Robust moving least-squares fitting with sharp features , 2005, ACM Trans. Graph..

[44]  Ingo Wald,et al.  Asynchronous BVH construction for ray tracing dynamic scenes on parallel multi-core architectures , 2007, EGPGV '07.

[45]  Bin Wang,et al.  The flexible global ocean-atmosphere-land system model, Grid-point Version 2: FGOALS-g2 , 2013, Advances in Atmospheric Sciences.

[46]  J. Z. Zhu,et al.  The superconvergent patch recovery and a posteriori error estimates. Part 1: The recovery technique , 1992 .

[47]  Jay Walter Larson,et al.  M × N Communication and Parallel Interpolation in Community Climate System Model Version 3 Using the Model Coupling Toolkit , 2005, Int. J. High Perform. Comput. Appl..

[48]  Cecelia DeLuca,et al.  Design and Implementation of Components in the Earth System Modeling Framework , 2005, Int. J. High Perform. Comput. Appl..

[49]  Jonathan Richard Shewchuk,et al.  Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator , 1996, WACG.

[50]  Timothy J. Tautges,et al.  MOAB : a mesh-oriented database. , 2004 .

[51]  Iulian Grindeanu,et al.  SIGMA Release v1.2 - Capabilities, Enhancements and Fixes , 2015 .

[52]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[53]  Patrick E. Farrell,et al.  Conservative interpolation between volume meshes by local Galerkin projection , 2011 .

[54]  Martin J. Gander,et al.  Algorithm 932: PANG: Software for nonmatching grid projections in 2D and 3D with linear complexity , 2013, TOMS.

[55]  Norberto M. Nigro,et al.  Conservative handling of arbitrary non-conformal interfaces using an efficient supermesh , 2017, J. Comput. Phys..

[56]  Miodrag Rančić.  An Efficient, Conservative, Monotonic Remapping for Semi-Lagrangian Transport Algorithms , 1995 .

[57]  Florian Lemarié,et al.  On the numerical stability of surface–atmosphere coupling in weather and climate models , 2016 .

[58]  R SlatteryStuart.  Mesh-free data transfer algorithms for partitioned multiphysics problems , 2016 .

[59]  Timothy J. Tautges,et al.  Mesh Interface Resolution and Ghost Exchange in a Parallel Mesh Representation , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum.