Is 1.7 x 10^10 Unknowns the Largest Finite Element System that Can Be Solved Today?

The hierarchical hybrid Grids (HHG) framework attempts to remove limitations on the size of problem that can be solved using a finite element discretization of a partial differential equation (PDE) by using a process of regular refinement, of an unstructured input grid, to generate a nested hierarchy of patch-wise structured grids that is suitable for use with geometric multigrid. The regularity of the resulting grids may be exploited in such a way that it is no longer necessary to explicitly assemble the global discretization matrix. In particular, given an appropriate input grid, the discretization matrix may be defined implicitly using stencils that are constant for each structured patch. This drastically reduces the amonnt of memory required for the discretization, thus allowing for a much larger problem to be solved. Here we present a brief description of the HHG framework: detailing the principles that led to solving a finite element system with 1.7 x 10^10 unknowns, on an SGI Altix supercomputer, using 1024 nodes, with an overall performance of 0.96 TFLOP/s, on a logically unstructured grid, using geometric mmiltigrid as a solver.

[1]  R. Rannacher,et al.  Asymptotic error expansion and Richardson extranpolation for linear finite elements , 1986 .

[2]  William L. Briggs,et al.  A multigrid tutorial , 1987 .

[3]  D. P. Young,et al.  A locally refined rectangular grid finite element method: application to computational fluid dynamics and computational physics , 1990 .

[4]  Richard Barrett,et al.  Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods , 1994, Other Titles in Applied Mathematics.

[5]  Ulrich Rüde,et al.  Implicit Extrapolation Methods for Multilevel Finite Element Computations , 1996, SIAM J. Sci. Comput..

[6]  Vipin Kumar,et al.  Parallel Multilevel k-way Partitioning Scheme for Irregular Graphs , 1996, Proceedings of the 1996 ACM/IEEE Conference on Supercomputing.

[7]  James Demmel,et al.  Making Sparse Gaussian Elimination Scalable by Static Pivoting , 1998, Proceedings of the IEEE/ACM SC98 Conference.

[8]  George Karypis,et al.  Multilevel k-way Partitioning Scheme for Irregular Graphs , 1998, J. Parallel Distributed Comput..

[9]  Ulrich Rüde,et al.  Implicit Extrapolation Methods for Variable Coefficient Problems , 1998, SIAM J. Sci. Comput..

[10]  M. Berger,et al.  Robust and efficient Cartesian mesh generation for component-based geometry , 1998 .

[11]  P. Colella,et al.  A Cartesian Grid Embedded Boundary Method for Poisson's Equation on Irregular Domains , 1998 .

[12]  James Demmel,et al.  A Supernodal Approach to Sparse Partial Pivoting , 1999, SIAM J. Matrix Anal. Appl..

[13]  William L. Briggs,et al.  A multigrid tutorial, Second Edition , 2000 .

[14]  Robert D. Falgout,et al.  hypre: A Library of High Performance Preconditioners , 2002, International Conference on Computational Science.

[15]  V. E. Henson,et al.  BoomerAMG: a parallel algebraic multigrid solver and preconditioner , 2002 .

[16]  A. ADoefaa,et al.  ? ? ? ? f ? ? ? ? ? , 2003 .

[17]  Ulrich Rüde,et al.  Hierarchical Hybrid Grids as Basis for Parallel Numerical Solution of PDE , 2003, Euro-Par.

[18]  Gerhard Wellein,et al.  Comparison of Parallel Programming Models on Clusters of SMP Nodes , 2003, HPSC.

[19]  Benjamin Karl Bergen,et al.  Hierarchical hybrid grids: data structures and core algorithms for multigrid , 2004, Numer. Linear Algebra Appl..

[20]  P. Colella,et al.  A node-centered local refinement algorithm for Poisson's equation in complex geometries , 2004 .

[21]  Mark F. Adams,et al.  Ultrascalable Implicit Finite Element Analyses in Solid Mechanics with over a Half a Billion Degrees of Freedom , 2004, Proceedings of the ACM/IEEE SC2004 Conference.

[22]  Ulrich Rüde,et al.  gridlib — A Parallel, Object-Oriented Framework for Hierarchical-Hybrid Grid Structures in Technical Simulation and Scientific Visualization , 2005 .

[23]  Arndt Bode,et al.  High Performance Computing In Science And Engineering, Munich 2004: Transactions Of The Second Joint HLRB And KONWIHR Status And Result Workshop, Marc ... University Of Munich, And Leibniz-Rechenzent , 2005 .

[24]  J. Bey,et al.  Tetrahedral grid refinement , 1995, Computing.