Towards a scalable fully-implicit fully-coupled resistive MHD formulation with stabilized FE methods

This paper explores the development of a scalable, nonlinear, fully-implicit stabilized unstructured finite element (FE) capability for 2D incompressible (reduced) resistive MHD. The discussion considers the implementation of a stabilized FE formulation in context of a fully-implicit time integration and direct-to-steady-state solution capability. The nonlinear solver strategy employs Newton-Krylov methods, which are preconditioned using fully-coupled algebraic multilevel preconditioners. These preconditioners are shown to enable a robust, scalable and efficient solution approach for the large-scale sparse linear systems generated by the Newton linearization. Verification results demonstrate the expected order-of-accuracy for the stabilized FE discretization. The approach is tested on a variety of prototype problems, including both low-Lundquist number (e.g., an MHD Faraday conduction pump and a hydromagnetic Rayleigh-Bernard linear stability calculation) and moderately-high Lundquist number (magnetic island coalescence problem) examples. Initial results that explore the scaling of the solution methods are presented on up to 4096 processors for problems with up to 64M unknowns on a CrayXT3/4. Additionally, a large-scale proof-of-capability calculation for 1 billion unknowns for the MHD Faraday pump problem on 24,000 cores is presented.

[1]  L Chacón,et al.  Scalable parallel implicit solvers for 3D magnetohydrodynamics , 2008 .

[2]  Pavel B. Bochev,et al.  A Taxonomy of Consistently Stabilized Finite Element Methods for the Stokes Problem , 2004, SIAM J. Sci. Comput..

[3]  Andrea Toselli,et al.  Convergence of some two-level overlapping domain decomposition preconditioners with smoothed aggregation coarse space , 2001 .

[4]  S. Wille,et al.  A preconditioned alternating inner-outer iterative solution method for the mixed finite element formulation of the Navier-Stokes equations , 1994 .

[5]  A. Gosman,et al.  Solution of the implicitly discretised reacting flow equations by operator-splitting , 1986 .

[6]  Chia-Jung Hsu.  Numerical Heat Transfer and Fluid Flow , 1981 .

[7]  J. Oden,et al.  A Posteriori Error Estimation in Finite Element Analysis , 2000 .

[8]  Homer F. Walker,et al.  Globally Convergent Inexact Newton Methods , 1994, SIAM J. Optim..

[9]  T. Hughes,et al.  Streamline upwind/Petrov-Galerkin formulations for convection dominated flows with particular emphasis on the incompressible Navier-Stokes equations , 1990 .

[10]  Thomas J. R. Hughes,et al.  A new finite element formulation for computational fluid dynamics: III. The generalized streamline operator for multidimensional advective-diffusive systems , 1986 .

[11]  Uri M. Ascher,et al.  Computer methods for ordinary differential equations and differential-algebraic equations , 1998 .

[12]  Rolf Rannacher,et al.  On the efficiency and robustness of implicit methods in computational astrophysics , 2001 .

[13]  Elaine S. Oran,et al.  Numerical Simulation of Reactive Flow , 1987 .

[14]  Steven J. Plimpton,et al.  Nonlinear magnetohydrodynamics simulation using high-order finite elements , 2004 .

[15]  Linda R. Petzold,et al.  Numerical solution of initial-value problems in differential-algebraic equations , 1996, Classics in applied mathematics.

[16]  J. Breslau,et al.  Nonlinear simulation studies of tokamaks and STs , 2003 .

[17]  John N. Shadid,et al.  Analysis of gallium arsenide deposition in a horizontal chemical vapor deposition reactor using massively parallel computations , 1999 .

[18]  Thomas M. Antonsen,et al.  Nonlinear reduced fluid equations for toroidal plasmas , 1984 .

[19]  D. A. Knoll,et al.  Coalescence of magnetic islands, sloshing, and the pressure problem , 2006 .

[20]  Eric Ronald Priest,et al.  Magnetic Reconnection: MHD Theory and Applications , 2000 .

[21]  Paul Lin,et al.  Performance of fully coupled domain decomposition preconditioners for finite element transport/reaction simulations , 2005 .

[22]  R. Codina.  Comparison of some finite element methods for solving the diffusion-convection-reaction equation , 1998 .

[23]  Louis A. Romero,et al.  Bifurcation Tracking Algorithms and Software for Large Scale Applications , 2005, Int. J. Bifurc. Chaos.

[24]  Jean-Frédéric Gerbeau,et al.  A stabilized finite element method for the incompressible magnetohydrodynamic equations , 2000, Numerische Mathematik.

[25]  R. Keppens,et al.  Implicit and semi-implicit schemes in the Versatile Advection Code : numerical tests , 1998 .

[26]  L. Driel-Gesztelyi.  An Introduction to Magnetohydrodynamics , 2004 .

[27]  R. Glowinski,et al.  Stabilized Finite Element Formulations for Incompressible Flow , 2003 .

[28]  D. Spalding,et al.  A calculation procedure for heat, mass and momentum transfer in three-dimensional parabolic flows , 1972 .

[29]  Richard B. Lehoucq,et al.  Large‐scale eigenvalue calculations for stability analysis of steady flows on massively parallel computers , 2001 .

[30]  G. Tóth.  The ∇·B=0 Constraint in Shock-Capturing Magnetohydrodynamics Codes , 2000 .

[31]  Marian Brezina,et al.  Convergence of algebraic multigrid based on smoothed aggregation , 1998, Numerische Mathematik.

[32]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[33]  Douglas S. Harned,et al.  Accurate semi-implicit treatment of the hall effect in magnetohydrodynamic computations , 1989 .

[34]  Margherita Pagani,et al.  Second Edition , 2004 .

[35]  Mark Frederick Hoemmen,et al.  An Overview of Trilinos , 2003 .

[36]  R. Codina.  Pressure Stability in Fractional Step Finite Element Methods for Incompressible Flows , 2001 .

[37]  T. Hughes,et al.  A new finite element formulation for computational fluid dynamics: V. Circumventing the Babuscka-Brezzi condition: A stable Petrov-Galerkin formulation of , 1986 .

[38]  J. Jackson,et al.  Classical Electrodynamics, 2nd Edition , 1975 .

[39]  Michel Visonneau,et al.  A new fully coupled solution of the Navier‐Stokes equations , 1994 .

[40]  William Gropp,et al.  Convergence rate estimate for a domain decomposition method , 1992 .

[41]  A. Aydemir,et al.  An implicit algorithm for compressible three-dimensional magnetohydrodynamic calculations , 1985 .

[42]  C. Vincent,et al.  A preconditioned conjugate gradient Uzawa-type method for the solution of the stokes problem by mixed Q1-P0 stabilized finite elements , 1992 .

[43]  Dieter Biskamp,et al.  Magnetic reconnection in plasmas , 2000 .

[44]  Homer F. Walker,et al.  An Inexact Newton Method for Fully Coupled Solution of the Navier-Stokes Equations with Heat and Mass Transport , 1997 .

[45]  T. Hughes.  Multiscale phenomena: Green's functions, the Dirichlet-to-Neumann formulation, subgrid scale models, bubbles and the origins of stabilized methods , 1995 .

[46]  Otto Eric Strack,et al.  ALEGRA : an arbitrary Lagrangian-Eulerian multimaterial, multiphysics code. , 2008 .

[47]  Louis A. Romero,et al.  Computational design and analysis of MOVPE reactors , 2001 .

[48]  Marian Brezina,et al.  Algebraic multigrid by smoothed aggregation for second and fourth order elliptic problems , 2005, Computing.

[49]  M. Benzi.  Preconditioning techniques for large linear systems: a survey , 2002 .

[50]  J. N. Shadid,et al.  A Fully-coupled Newton-Krylov Solution Method for Parallel Unstructured Finite Element Fluid Flow, Heat and Mass Transfer Simulations , 1999 .

[51]  Paul T. Lin,et al.  Performance of a parallel algebraic multilevel preconditioner for stabilized finite element semiconductor device modeling , 2009, J. Comput. Phys..

[52]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[53]  F. Shakib.  Finite element analysis of the compressible Euler and Navier-Stokes equations , 1989 .

[54]  Leonard J. Gray,et al.  Numerical simulation of non-viscous liquid pinch-off using a coupled level set-boundary integral method , 2009, J. Comput. Phys..

[55]  A. Quarteroni,et al.  Factorization methods for the numerical approximation of Navier-Stokes equations , 2000 .

[56]  Barry Smith,et al.  Domain Decomposition Methods for Partial Differential Equations , 1997 .

[57]  William B. Thompson.  Transport Processes in the Plasma , 1960 .

[58]  H. R. Strauss,et al.  Nonlinear, three‐dimensional magnetohydrodynamics of noncircular tokamaks , 1976 .

[59]  John N. Shadid,et al.  On a multilevel preconditioning module for unstructured mesh Krylov solvers: two-level Schwarz , 2002 .

[60]  Koulis Pericleous,et al.  The numerical modelling of DC electromagnetic pump and brake flow , 1995 .

[61]  Dana A. Knoll,et al.  An Implicit, Nonlinear Reduced Resistive MHD Solver , 2002 .

[62]  M. Mcpherson,et al.  Introduction to fluid mechanics , 1997 .

[63]  D. A. Dunnett.  Classical Electrodynamics , 2020, Nature.

[64]  Homer F. Walker,et al.  Globalization Techniques for Newton-Krylov Methods and Applications to the Fully Coupled Solution of the Navier-Stokes Equations , 2006, SIAM Rev..

[65]  T. Hughes,et al.  A new finite element formulation for computational fluid dynamics: VI. Convergence analysis of the generalized SUPG formulation for linear time-dependent multi-dimensional advective-diffusive systems , 1987 .

[66]  Philip M. Gresho,et al.  On the theory of semi‐implicit projection methods for viscous incompressible flow and its implementation via a finite element method that also introduces a nearly consistent mass matrix. Part 1: Theory , 1990 .

[67]  Richard D Hazeltine,et al.  A four‐field model for tokamak plasma dynamics , 1985 .

[68]  Ramon Codina,et al.  Stabilized Finite Element Approximation of the Stationary Magneto-Hydrodynamics Equations , 2006 .

[69]  Dalton D. Schnack,et al.  Semi-implicit magnetohydrodynamic calculations , 1987 .

[70]  C. Munz,et al.  Hyperbolic divergence cleaning for the MHD equations , 2002 .

[71]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[72]  James A. Sethian,et al.  THE DERIVATION AND NUMERICAL SOLUTION OF THE EQUATIONS FOR ZERO MACH NUMBER COMBUSTION , 1985 .

[73]  John C. Strikwerda,et al.  The Accuracy of the Fractional Step Method , 1999, SIAM J. Numer. Anal..

[74]  Rony Keppens,et al.  Implicit and semi-implicit schemes: Algorithms , 1999 .

[75]  A. Huerta,et al.  Finite Element Methods for Flow Problems , 2003 .

[76]  Gábor Tóth.  Versatile Advection Code , 1997, HPCN Europe.

[77]  Dalton D. Schnack,et al.  Semi-implicit method for long time scale magnetohydrodynamic computations in three dimensions , 1986 .

[78]  Carol S. Woodward,et al.  Operator-Based Preconditioning of Stiff Hyperbolic Systems , 2010, SIAM J. Sci. Comput..

[79]  Jonathan J. Hu,et al.  ML 5.0 Smoothed Aggregation Users's Guide , 2006 .

[80]  John N. Shadid,et al.  An Improved Convergence Bound for Aggregation-Based Domain Decomposition Preconditioners , 2005, SIAM J. Matrix Anal. Appl..

[81]  Tamara G. Kolda,et al.  An overview of the Trilinos project , 2005, TOMS.

[82]  D. A. Knoll,et al.  A 2D high-ß Hall MHD implicit nonlinear solver , 2003 .

[83]  R. A. Wentzell,et al.  Hydrodynamic and Hydromagnetic Stability. By S. CHANDRASEKHAR. Clarendon Press: Oxford University Press, 1961. 652 pp. £5. 5s. , 1962, Journal of Fluid Mechanics.

[84]  S. Orszag,et al.  High-order splitting methods for the incompressible Navier-Stokes equations , 1991 .

[85]  Michel Fortin,et al.  A conservative stabilized finite element method for the magneto-hydrodynamic equations , 1999 .

[86]  Max Gunzburger,et al.  Finite Element Methods for Viscous Incompressible Flows: A Guide to Theory, Practice, and Algorithms , 1989 .

[87]  A. Chorin.  A Numerical Method for Solving Incompressible Viscous Flow Problems , 1997 .

[88]  Homer F. Walker,et al.  Choosing the Forcing Terms in an Inexact Newton Method , 1996, SIAM J. Sci. Comput..

[89]  Shrinivas Lankalapalli,et al.  An adaptive finite element method for magnetohydrodynamics , 1998, J. Comput. Phys..

[90]  Dinshaw Balsara,et al.  Divergence-free adaptive mesh refinement for Magnetohydrodynamics , 2001 .

[91]  Paul Lin,et al.  Performance of a Petrov–Galerkin algebraic multilevel preconditioner for finite element modeling of the semiconductor device drift‐diffusion equations , 2010 .

[92]  S. C. Jardin,et al.  Implicit Solution of the Four-field Extended-magnetohydroynamic Equations using High-order High-continuity Finite Elements , 2004 .

[93]  Sandia Report.  Amesos 2.0 Reference Guide , 2004 .

[94]  Paul R. Woodward,et al.  On the Divergence-free Condition and Conservation Laws in Numerical Simulations for Supersonic Magnetohydrodynamical Flows , 1998 .

[95]  William L. Briggs,et al.  A multigrid tutorial, Second Edition , 2000 .

[96]  F. Brezzi.  On the existence, uniqueness and approximation of saddle-point problems arising from lagrangian multipliers , 1974 .

[97]  John N. Shadid,et al.  Computational Analysis and Optimization of a Chemical Vapor Deposition Reactor with Large-Scale Computing , 2004 .

[98]  Jonathan J. Hu,et al.  ML 3.1 smoothed aggregation user's guide. , 2004 .

[99]  Tamara G. Kolda,et al.  Graph partitioning models for parallel computing , 2000, Parallel Comput..

[100]  Jean-Luc Guermond,et al.  International Journal for Numerical Methods in Fluids on Stability and Convergence of Projection Methods Based on Pressure Poisson Equation , 2022 .

[101]  David E. Keyes,et al.  Additive Schwarz-based fully coupled implicit methods for resistive Hall magnetohydrodynamic problems , 2007, J. Comput. Phys..

[102]  T. Hughes,et al.  The Galerkin/least-squares method for advective-diffusive equations , 1988 .

[103]  Stephen C. Jardin,et al.  Implicit solution of the four-field extended-magnetohydrodynamic equations using high-order high-continuity finite elementsa) , 2004 .

[104]  Stefaan Poedts,et al.  Principles of Magnetohydrodynamics: With Applications to Laboratory and Astrophysical Plasmas , 2004 .

[105]  Wolfgang Kerner,et al.  Semi-implicit method for three-dimensional compressible magnetohydrodynamic simulation☆ , 1985 .

[106]  Barry F. Smith,et al.  Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations , 1996 .

[107]  Paul Lin,et al.  Performance of fully coupled algebraic multilevel domain decomposition preconditioners for incompressible flow and transport , 2006 .

[108]  Dieter Biskamp,et al.  Coalescence of Magnetic Islands , 1980 .

[109]  William Gropp,et al.  High-performance parallel implicit CFD , 2001, Parallel Comput..

[110]  N. N. Komarov,et al.  Self-focusing of local plasma currents , 1965 .

[111]  Allen C. Robinson,et al.  Three-dimensional z-pinch wire array modeling with ALEGRA-HEDP , 2003, Comput. Phys. Commun..

[112]  H. K. Moffatt.  Magnetic Field Generation in Electrically Conducting Fluids , 1978 .

[113]  Yousef Saad,et al.  Hybrid Krylov Methods for Nonlinear Systems of Equations , 1990, SIAM J. Sci. Comput..

[114]  A. Hujeirat,et al.  IRMHD: an implicit radiative and magnetohydrodynamical solver for self-gravitating systems , 1998 .

[115]  Paul R. Woodward,et al.  A Simple Finite Difference Scheme for Multidimensional Magnetohydrodynamical Equations , 1998 .

[116]  Ray S. Tuminaro,et al.  A New Petrov--Galerkin Smoothed Aggregation Preconditioner for Nonsymmetric Linear Systems , 2008, SIAM J. Sci. Comput..

[117]  Michel Visonneau,et al.  A new fully coupled method for computing turbulent flows , 2001 .

[118]  O. Axelsson.  Iterative solution methods , 1995 .

[119]  W. Habashi,et al.  A finite element method for magnetohydrodynamics , 2001 .

[120]  Carol S. Woodward,et al.  A fully implicit numerical method for single-fluid resistive magnetohydrodynamics , 2006, J. Comput. Phys..

[121]  J. Oden,et al.  A Posteriori Error Estimation in Finite Element Analysis: Oden/A Posteriori , 2000 .

[122]  Courtenay T. Vaughan,et al.  Zoltan data management services for parallel dynamic applications , 2002, Comput. Sci. Eng..

[123]  John N. Shadid,et al.  Aztec user`s guide. Version 1 , 1995 .

[124]  L. Margolin,et al.  On balanced approximations for time integration of multiple time scale systems , 2003 .

[125]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[126]  R. Pellat,et al.  Coalescence of magnetic islands , 1983 .

[127]  Bruce Hendrickson,et al.  Load balancing fictions, falsehoods and fallacies☆ , 2000 .

[128]  T. Hughes,et al.  A new finite element formulation for computational fluid dynamics: II. Beyond SUPG , 1986 .

[129]  P. Colella,et al.  A second-order projection method for the incompressible navier-stokes equations , 1989 .

[130]  D. Balsara,et al.  A Staggered Mesh Algorithm Using High Order Godunov Fluxes to Ensure Solenoidal Magnetic Fields in Magnetohydrodynamic Simulations , 1999 .

[131]  Francesco Miniati,et al.  A Divergence-free Upwind Code for Multidimensional Magnetohydrodynamic Flows , 1998 .

[132]  José Roberto Cardoso,et al.  Three-dimensional finite element analysis of MHD duct flow by the penalty function formulation , 2001 .

[133]  L. Chacón,et al.  A non-staggered, conservative, V×B=0' finite-volume scheme for 3D implicit extended magnetohydrodynamics in curvilinear geometries , 2004, Comput. Phys. Commun..

[134]  Yousef Saad,et al.  Convergence Theory of Nonlinear Newton-Krylov Algorithms , 1994, SIAM J. Optim..

[135]  T. Hughes,et al.  A theoretical framework for Petrov-Galerkin methods with discontinuous weighting functions: application to the streamline-upwind procedure. , 1982 .

[136]  Alexandre Ern,et al.  Nonlinear diffusion and discrete maximum principle for stabilized Galerkin approximations of the convection-diffusion-reaction equation , 2002 .

[137]  Vipin Kumar,et al.  Parallel Multilevel k-way Partitioning Scheme for Irregular Graphs , 1996, Proceedings of the 1996 ACM/IEEE Conference on Supercomputing.

[138]  J. B. Perot,et al.  An analysis of the fractional step method , 1993 .

[139]  Gregory W. Brown,et al.  Mesh partitioning for implicit computations via iterative domain decomposition: Impact and optimization of the subdomain aspect ratio , 1995 .

[140]  Luis Chacon,et al.  An optimal, parallel, fully implicit Newton–Krylov solver for three-dimensional viscoresistive magnetohydrodynamicsa) , 2008 .

[141]  Timothy A. Davis,et al.  Direct methods for sparse linear systems , 2006, Fundamentals of algorithms.

[142]  J. W. Ruge,et al.  4. Algebraic Multigrid , 1987 .

[143]  R. Lehoucq,et al.  Computational bifurcation and stability studies of the 8: 1 thermal cavity problem , 2002 .

[144]  John N. Shadid,et al.  Fundamental models of the metalorganic vapor-phase epitaxy of gallium nitride and their use in reactor design , 2000 .

[145]  Paul Lin,et al.  Large-scale stabilized FE computational analysis of nonlinear steady state transport/reaction systems. , 2004 .