Tusas: A fully implicit parallel approach for coupled phase-field equations

We develop a fully-coupled, fully-implicit approach for phase-field modeling of solidification in metals and alloys. Predictive simulation of solidification in pure metals andmetal alloys remains a significant challenge in the field of materials science, as microstructure formation during the solidification process plays a critical role in the properties and performance of the solid material. Our simulation approach consists of a finite element spatial discretization of the fully-coupled nonlinear system of partial differential equations at the microscale, which is treated implicitly in time with a preconditioned Jacobian-free Newton-Krylov method. The approach is algorithmically scalable as well as efficient due to an effective preconditioning strategy based on algebraic multigrid and block factorization. We implement this approach in the open-source Tusas framework, which is a general, flexible tool developed in C++ for solving coupled systems of nonlinear partial differential equations. The performance of our approach is analyzed in terms of algorithmic scalability and efficiency, while the computational performance of Tusas is presented in terms of parallel scalability and efficiency on emerging heterogeneous architectures. We demonstrate that modern algorithms, discretizations, and computational science, and heterogeneous hardware provide a robust route for predictive phasefield simulation of microstructure evolution during additive manufacturing.

[1]  Satoshi Matsuoka,et al.  Peta-scale phase-field simulation for dendritic solidification on the TSUBAME 2.0 supercomputer , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[2]  Inanc Senocak,et al.  An MPI-CUDA Implementation for Massively Parallel Incompressible Flow Computations on Multi-GPU Clusters , 2010 .

[3]  Katherine Yelick,et al.  Exascale applications: skin in the game , 2020, Philosophical Transactions of the Royal Society A.

[4]  H. Matthies,et al.  Partitioned Strong Coupling Algorithms for Fluid-Structure-Interaction , 2003 .

[5]  Yongpeng Zhang,et al.  A GPU accelerated aggregation algebraic multigrid method , 2014, Comput. Math. Appl..

[6]  Vaibhav Joshi,et al.  An adaptive variational procedure for the conservative and positivity preserving Allen-Cahn phase-field model , 2017, J. Comput. Phys..

[7]  Manfred Liebmann,et al.  A Parallel Algebraic Multigrid Solver on Graphics Processing Units , 2009, HPCA.

[8]  L. R. Scott,et al.  The Mathematical Theory of Finite Element Methods , 1994 .

[9]  Tahany Ibrahim El-Wardany,et al.  Phase Field Simulations of Microstructure Evolution in IN718 using a Surrogate Ni–Fe–Nb Alloy during Laser Powder Bed Fusion , 2018, Metals.

[10]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[11]  William L. Briggs,et al.  A multigrid tutorial , 1987 .

[12]  K. Thornton,et al.  PRISMS-PF: A general framework for phase-field modeling with a matrix-free finite element method , 2020, npj Computational Materials.

[13]  Daniel Sunderland,et al.  Kokkos: Enabling manycore performance portability through polymorphic memory access patterns , 2014, J. Parallel Distributed Comput..

[14]  J. Lambert Numerical Methods for Ordinary Differential Systems: The Initial Value Problem , 1991 .

[15]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[16]  Heike Emmerich,et al.  Phase-field modeling of microstructure formation during rapid solidification in Inconel 718 superalloy , 2015 .

[17]  Derek Gaston,et al.  MOOSE: A parallel computational framework for coupled systems of nonlinear equations , 2009 .

[18]  A. Karma Phase-field formulation for quantitative modeling of alloy solidification. , 2001, Physical review letters.

[19]  Vaibhav Joshi,et al.  A positivity preserving and conservative variational scheme for phase-field modeling of two-phase flows , 2017, J. Comput. Phys..

[20]  Li Ma,et al.  On the primary spacing and microsegregation of cellular dendrites in laser deposited Ni–Nb alloys , 2017, 1705.06669.

[21]  Vaibhav Joshi,et al.  A hybrid variational Allen‐Cahn/ALE scheme for the coupled analysis of two‐phase fluid‐structure interaction , 2018, International Journal for Numerical Methods in Engineering.

[22]  Homer F. Walker,et al.  Globalization Techniques for Newton-Krylov Methods and Applications to the Fully Coupled Solution of the Navier-Stokes Equations , 2006, SIAM Rev..

[23]  Ranadip Acharya,et al.  Prediction of microstructure in laser powder bed fusion process , 2017 .

[24]  Christina Freytag,et al.  Using Mpi Portable Parallel Programming With The Message Passing Interface , 2016 .

[25]  Homer F. Walker,et al.  Choosing the Forcing Terms in an Inexact Newton Method , 1996, SIAM J. Sci. Comput..

[26]  D. Keyes,et al.  Jacobian-free Newton-Krylov methods: a survey of approaches and applications , 2004 .

[27]  Alexander Düster,et al.  A partitioned solution approach for electro-thermo-mechanical problems , 2015 .

[28]  Gupta,et al.  Calculus Of Variations With Applications , 2004 .

[29]  Pak Lui,et al.  Strong scaling of general-purpose molecular dynamics simulations on GPUs , 2014, Comput. Phys. Commun..

[30]  Richard C. Martineau,et al.  On physics-based preconditioning of the Navier-Stokes equations , 2009, J. Comput. Phys..

[31]  Supriyo Ghosh,et al.  Predictive modeling of solidification during laser additive manufacturing of nickel superalloys: recent developments, future directions , 2017, 1707.09292.

[32]  Zhangxin Chen,et al.  Accelerating algebraic multigrid solvers on NVIDIA GPUs , 2015, Comput. Math. Appl..

[33]  T. Takaki,et al.  GPU-accelerated 3D phase-field simulations of dendrite competitive growth during directional solidification of binary alloy , 2015 .

[34]  D. Dinkler,et al.  A monolithic approach to fluid–structure interaction using space–time finite elements , 2004 .

[35]  A. Karma,et al.  Quantitative phase-field model of alloy solidification. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[36]  Wolfgang Hackbusch,et al.  Multi-grid methods and applications , 1985, Springer series in computational mathematics.

[37]  Carol S. Woodward,et al.  Enabling New Flexibility in the SUNDIALS Suite of Nonlinear and Differential/Algebraic Equation Solvers , 2020, ACM Trans. Math. Softw..

[38]  Kevin McReynolds,et al.  Simulation of temperature, stress and microstructure fields during laser deposition of Ti–6Al–4V , 2018, Modelling and simulation in materials science and engineering.

[39]  Li Ma,et al.  Application of Finite Element, Phase-field, and CALPHAD-based Methods to Additive Manufacturing of Ni-based Superalloys. , 2017, Acta materialia.

[40]  Sivasankaran Rajamanickam,et al.  Amesos2 and Belos: Direct and iterative solvers for large sparse linear systems , 2012, Sci. Program..

[41]  Li Ma,et al.  Single-Track Melt-Pool Measurements and Microstructures in Inconel 625 , 2018, 1802.05827.

[42]  William J. Rider,et al.  A Multigrid Preconditioned Newton-Krylov Method , 1999, SIAM J. Sci. Comput..

[43]  Michael A. Heroux AztecOO user guide. , 2004 .

[44]  Mark R. Stoudt,et al.  Formation of Nb-rich droplets in laser deposited Ni-matrix microstructures , 2017, 1710.11525.

[45]  G. Caginalp,et al.  Stefan and Hele-Shaw type models as asymptotic limits of the phase-field equations. , 1989, Physical review. A, General physics.

[46]  Stephen Lee,et al.  Exascale Computing in the United States , 2019, Computing in Science & Engineering.

[47]  B. Blanpain,et al.  An introduction to phase-field modeling of microstructure evolution , 2008 .

[48]  Y. Ji,et al.  Understanding Microstructure Evolution During Additive Manufacturing of Metallic Alloys Using Phase-Field Modeling , 2018 .

[49]  L. Margolin,et al.  On balanced approximations for time integration of multiple time scale systems , 2003 .

[50]  Homer F. Walker,et al.  NITSOL: A Newton Iterative Solver for Nonlinear Systems , 1998, SIAM J. Sci. Comput..

[51]  Kessler,et al.  Velocity selection in dendritic growth. , 1986, Physical review. B, Condensed matter.

[52]  Luis Chacón,et al.  Jacobian–Free Newton–Krylov Methods for the Accurate Time Integration of Stiff Wave Systems , 2005, J. Sci. Comput..

[53]  Dana A. Knoll,et al.  An Implicit Nonlinearly Consistent Method for the Two-Dimensional Shallow-Water Equations with Coriolis Force , 2002 .

[54]  A. Kokabi,et al.  Phase-field simulation of weld solidification microstructure in an Al–Cu alloy , 2008 .

[55]  Long-Qing Chen Phase-Field Models for Microstructure Evolution , 2002 .

[56]  van Eh Harald Brummelen,et al.  A monolithic approach to fluid–structure interaction , 2004 .

[57]  Qingyan Xu,et al.  GPU-accelerated three-dimensional phase-field simulation of dendrite growth in a nickel-based superalloy , 2017 .

[58]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[59]  Gordon Erlebacher,et al.  High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster , 2010, J. Comput. Phys..

[60]  W. Kurz,et al.  Fundamentals of Solidification , 1990 .

[61]  J. E. Hilliard,et al.  Free Energy of a Nonuniform System. I. Interfacial Free Energy , 1958 .

[62]  Robert F. Sekerka,et al.  Morphology: from sharp interface to phase field models , 2004 .

[63]  Dana A. Knoll,et al.  An Implicit, Nonlinear Reduced Resistive MHD Solver , 2002 .

[64]  A. Karma,et al.  Phase-Field Simulation of Solidification , 2002 .

[65]  T. Blacker,et al.  Modeling of additive manufacturing processes for metals: Challenges and opportunities , 2017 .

[66]  Alaa Elwany,et al.  Finite Interface Dissipation Phase Field Modeling of Ni-Nb Under Additive Manufacturing Conditions , 2019, Acta Materialia.

[67]  Robert D. Falgout,et al.  hypre: A Library of High Performance Preconditioners , 2002, International Conference on Computational Science.

[68]  James A. Warren,et al.  FiPy: Partial Differential Equations with Python , 2009, Computing in Science & Engineering.

[69]  Russ Rew,et al.  NetCDF: an interface for scientific data access , 1990, IEEE Computer Graphics and Applications.

[70]  L. Dagum,et al.  OpenMP: an industry standard API for shared-memory programming , 1998 .

[71]  J. Cahn,et al.  A microscopic theory for antiphase boundary motion and its application to antiphase domain coasening , 1979 .

[72]  Olle Heinonen,et al.  Phase field benchmark problems for dendritic growth and linear elasticity , 2018, Computational Materials Science.

[73]  Tamara G. Kolda,et al.  An overview of the Trilinos project , 2005, TOMS.

[74]  T. Takaki,et al.  GPU-accelerated phase-field simulation of dendritic solidification in a binary alloy , 2011 .

[75]  M. Clemens,et al.  GPU Acceleration of Algebraic Multigrid Preconditioners for Discrete Elliptic Field Problems , 2014, IEEE Transactions on Magnetics.

[76]  I. Steinbach Phase-field models in materials science , 2009 .

[77]  Milo R. Dorr,et al.  A numerical algorithm for the solution of a phase-field model of polycrystalline materials , 2008, J. Comput. Phys..

[78]  Nana Ofori-Opoku,et al.  Simulation and analysis of γ-Ni cellular growth during laser powder deposition of Ni-based superalloys. , 2017, Computational materials science.

[79]  Yousef Saad,et al.  Hybrid Krylov Methods for Nonlinear Systems of Equations , 1990, SIAM J. Sci. Comput..

[80]  Li Feng,et al.  Multi-GPU hybrid programming accelerated three-dimensional phase-field model in binary alloy , 2018 .