Hybrid Parallel Multigrid Methods for Geodynamical Simulations

Even on modern supercomputer architectures, Earth mantle simulations are so compute intensive that they are considered grand challenge applications. The dominating roadblocks in this branch of Geophysics are model complexity and uncertainty in parameters and data, e.g., rheology and seismically imaged mantle heterogeneity, as well as the enormous space and time scales that must be resolved in the computational models. This article reports on a massively parallel all-at-once multigrid solver for the Stokes system as it arises in mantle convection models. The solver employs the hierarchical hybrid grids framework and demonstrates that a system with coupled velocity components and with more than a trillion (1. 7 ⋅ 1012) degrees of freedom can be solved in about 1,000 s using 40,960 compute cores of JUQUEEN. The simulation framework is used to investigate the influence of asthenosphere thickness and viscosity on upper mantle velocities in a static scenario. Additionally, results for a time-dependent simulation with a time-variable temperature-dependent viscosity model are presented.

[1]  Walter R. Roest,et al.  Age, spreading rates, and spreading asymmetry of the world's ocean crust , 2008 .

[2]  Georg Stadler,et al.  Large-scale adaptive mantle convection simulation , 2013 .

[3]  A. Brandt,et al.  Multigrid Solutions to Elliptic Flow Problems , 1979 .

[4]  Achi Brandt,et al.  Multigrid Techniques: 1984 Guide with Applications to Fluid Dynamics, Revised Edition , 2011 .

[5]  J. Douglas,et al.  Stabilized mixed methods for the Stokes problem , 1988 .

[6]  Lars Stixrude,et al.  Thermodynamics of mantle minerals – I. Physical properties , 2005 .

[7]  McSween Hy,et al.  Evidence for Life in a Martian Meteorite , 1997 .

[8]  André Garon,et al.  Weak imposition of the slip boundary condition on curved boundaries for Stokes flow , 2014, J. Comput. Phys..

[9]  Ulrich Rüde,et al.  A Massively Parallel Multigrid Method for Finite Elements , 2006, Computing in Science & Engineering.

[10]  Barbara I. Wohlmuth,et al.  Local Mass-Corrections for Continuous Pressure Approximations of Incompressible Flow , 2014, SIAM J. Numer. Anal..

[11]  Martin Kronbichler,et al.  Algorithms and data structures for massively parallel generic adaptive finite element codes , 2011, ACM Trans. Math. Softw..

[12]  Vivette Girault,et al.  Finite Element Methods for Navier-Stokes Equations - Theory and Algorithms , 1986, Springer Series in Computational Mathematics.

[13]  Hari Sundar,et al.  Comparison of multigrid algorithms for high‐order continuous finite element discretizations , 2014, Numer. Linear Algebra Appl..

[14]  Adrian Lenardic,et al.  Three‐dimensional mantle convection simulations with a low‐viscosity asthenosphere and the relationship between heat flow and the horizontal length scale of convection , 2008 .

[15]  C. R. Hagelberg,et al.  Mantle circulation models with variational data assimilation: inferring past mantle flow and structure from plate motion histories and seismic tomography , 2001 .

[16]  J. Mitrovica,et al.  Haskell [1935] revisited , 1996 .

[17]  John R. Baumgardner,et al.  Three-dimensional treatment of convective flow in the earth's mantle , 1985 .

[18]  Constantine Bekas,et al.  An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth's mantle , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[19]  Michel Fortin,et al.  Mixed and Hybrid Finite Element Methods , 2011, Springer Series in Computational Mathematics.

[20]  Joachim Schöberl,et al.  On Schwarz-type Smoothers for Saddle Point Problems , 2003, Numerische Mathematik.

[21]  R. Bank,et al.  A class of iterative methods for solving saddle point problems , 1989 .

[22]  Bramley J. Murton,et al.  A continuous 55-million-year record of transient mantle plume activity beneath Iceland , 2014 .

[23]  D. L. Anderson,et al.  Preliminary reference earth model , 1981 .

[24]  P. R. Vogt,et al.  Asthenosphere motion recorded by the ocean floor south of Iceland , 1971 .

[25]  Carsten Burstedde,et al.  p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees , 2011, SIAM J. Sci. Comput..

[26]  Yvan Notay,et al.  A Simple and Efficient Segregated Smoother for the Discrete Stokes Equations , 2014, SIAM J. Sci. Comput..

[27]  Sri Widiyantoro,et al.  Global seismic tomography: A snapshot of convection in the Earth: GSA Today , 1997 .

[28]  Nicholas J. White,et al.  Transient convective uplift of an ancient buried landscape , 2010 .

[29]  Barbara I. Wohlmuth,et al.  Resilience for Multigrid Software at the Extreme Scale , 2015, ArXiv.

[30]  Bernhard S. A. Schuberth,et al.  Reconciling dynamic and seismic models of Earth's lower mantle: The dominant role of thermal heterogeneity , 2012 .

[31]  Y. Ricard,et al.  Physics of Mantle Convection , 2007 .

[32]  Philip M. Gresho,et al.  The implementation of normal and/or tangential boundary conditions in finite element codes for incompressible fluid flow , 1982 .

[33]  Ulrich Rüde,et al.  Fast asthenosphere motion in high‐resolution global mantle flow models , 2015 .

[34]  Gerhard Wellein,et al.  Exploring performance and power properties of modern multi‐core chips via simple machine models , 2012, Concurr. Comput. Pract. Exp..

[35]  Jeannot Trampert,et al.  Using probabilistic seismic tomography to test mantle velocity–density relationships , 2003 .

[36]  Hendrik Jan van Heijst,et al.  Global transition zone tomography , 2004 .

[37]  Richards,et al.  Time scales and heterogeneous structure in geodynamic earth models , 1998, Science.

[38]  Barbara I. Wohlmuth,et al.  Mass-corrections for the conservative coupling of flow and transport on collocated meshes , 2016, J. Comput. Phys..

[39]  Barbara I. Wohlmuth,et al.  Performance and Scalability of Hierarchical Hybrid Multigrid Solvers for Stokes Systems , 2015, SIAM J. Sci. Comput..

[40]  Walter Zulehner,et al.  Analysis of iterative methods for saddle point problems: a unified approach , 2002, Math. Comput..

[41]  Howard C. Elman,et al.  Finite Elements and Fast Iterative Solvers: with Applications in Incompressible Fluid Dynamics , 2014 .

[42]  Lapo Boschi,et al.  A comparison of tomographic and geodynamic mantle models , 2002 .

[43]  Barbara Wohlmuth,et al.  Solution Techniques for the Stokes System: A priori and a posteriori modifications, resilient algorithms , 2015, 1511.05759.

[44]  Louis Moresi,et al.  Role of temperature‐dependent viscosity and surface plates in spherical shell models of mantle convection , 2000 .

[45]  David J. Stevenson,et al.  Effects of multiple phase transitions in a three-dimensional spherical model of convection in Earth's mantle , 1994 .

[46]  Andreas Fichtner,et al.  Full seismic waveform tomography for upper-mantle structure in the Australasian region using adjoint methods , 2009 .

[47]  P. Tackley,et al.  Mantle convection and plate tectonics: toward an integrated physical and chemical theory , 2000, Science.

[48]  R. Verfürth.  Finite element approximation on incompressible Navier-Stokes equations with slip boundary condition , 1987 .

[49]  Ulrich Rüde,et al.  Hierarchical hybrid grids: achieving TERAFLOP performance on large scale finite element simulations , 2007, Int. J. Parallel Emergent Distributed Syst..

[50]  Eh Tan,et al.  GeoFramework: Coupling multiple models of mantle convection within a computational framework , 2006 .

[51]  N. A. Haskell.  The Motion of a Viscous Fluid Under a Surface Load , 1935 .

[52]  Ulrich Rüde,et al.  Towards Textbook Efficiency for Parallel Multigrid , 2015 .

[53]  Mark A. Richards,et al.  A sensitivity study of three-dimensional spherical mantle convection at 108 Rayleigh number: Effects of depth-dependent viscosity, heating mode, and an endothermic phase change , 1997 .

[54]  Louis Moresi,et al.  A benchmark study on mantle convection in a 3‐D spherical shell using CitcomS , 2008 .

[55]  Paul J. Tackley,et al.  Effects of strongly variable viscosity on three‐dimensional compressible convection in planetary mantles , 1996 .

[56]  Barbara I. Wohlmuth,et al.  A quantitative performance analysis for Stokes solvers at the extreme scale , 2015, ArXiv.

[57]  Samuel Williams,et al.  Roofline: an insightful visual performance model for multicore architectures , 2009, CACM.

[58]  Maria Seton,et al.  Global continental and ocean basin reconstructions since 200 Ma , 2012 .

[59]  Perumal Nithiarasu,et al.  A hierarchical mesh refinement technique for global 3-D spherical mantle convection modelling , 2013 .

[60]  Hans-Peter Bunge,et al.  Cluster Design in the Earth Sciences Tethys , 2006, HPCC.

[61]  Hans-Peter Bunge,et al.  Mantle convection modeling on parallel virtual machines , 1995 .