Multi-scale CAFE Framework for Simulating Fracture in Heterogeneous Materials Implemented in Fortran Co-arrays and MPI

Fortran coarrays have been used as an extension to the standard for over 20 years, mostly on Cray systems. Their appeal to users increased substantially when they were standardised in 2010. In this work we show that coarrays offer simple and intuitive data structures for 3D cellular automata (CA) modelling of material microstructures. We show how coarrays can be used together with an MPI finite element (FE) library to create a two-way concurrent hierarchical and scalable multi-scale CAFE deformation and fracture framework. Design of a coarray cellular automata microstructure evolution library CGPACK is described. A highly portable MPI FE library ParaFEM was used in this work. We show that independently CGPACK and ParaFEM programs can scale up well into tens of thousands of cores. Strong scaling of a hybrid ParaFEM/CGPACK MPI/coarray multi-scale framework was measured on an important solid mechanics practical example of a fracture of a steel round bar under tension. That program did not scale beyond 7 thousand cores. Excessive synchronisation might be one contributing factor to relatively poor scaling. Therefore we conclude with a comparative analysis of synchronisation requirements in MPI and coarray programs. Specific challenges of synchronising a coarray library are discussed.

[1]  Lee Margetts,et al.  Scaling hybrid coarray/MPI miniapps on Archer , 2016 .

[2]  Lee Margetts,et al.  Sutured tendon repair; a multi-scale finite element model , 2015, Biomechanics and modeling in mechanobiology.

[3]  G. Guillemota,et al.  Interaction between single grain solidification and macrosegregation: Application of a cellular automaton—Finite element model , 2007 .

[4]  D. V. Griffiths,et al.  Spatial variability in the coefficient of thermal expansion induces pre-service stresses in computer models of virgin Gilsocarbon bricks , 2015 .

[5]  A Fanfarillo,et al.  Parallel programming techniques for heterogeneous exascale computing platforms , 2014 .

[6]  Ian C. Howard,et al.  The CAFE model of fracture—application to a TMCR steel , 2006 .

[7]  Mats Hamrud,et al.  A Partitioned Global Address Space implementation of the European Centre for Medium Range Weather Forecasts Integrated Forecasting System , 2015, Int. J. High Perform. Comput. Appl..

[8]  Lee Margetts,et al.  Evaluating the macroscopic yield behaviour of trabecular bone using a nonlinear homogenisation approach. , 2016, Journal of the mechanical behavior of biomedical materials.

[9]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[10]  Anton Shterenlikht,et al.  CELLULAR AUTOMATA MODELLING OF NANOCRYSTALLINE INSTABILITY * , 2012 .

[11]  Dan Nagle,et al.  OpenCoarrays: Open-source Transport Layers Supporting Coarray Fortran Compilers , 2014, PGAS.

[12]  Charles Moulinec,et al.  Performance of Parallel IO on ARCHER , 2015 .

[13]  D. V. Griffiths,et al.  Programming the finite element method , 1982 .

[14]  Toni Collis Using NETCDF with Fortran on ARCHER , 2016 .

[15]  Sumitesh Das,et al.  A general method for coupling microstructural response with structural performance , 2006, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[16]  Lee Margetts,et al.  Three-dimensional cellular automata modelling of cleavage propagation across crystal boundaries in polycrystalline microstructures , 2015, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[17]  Wei Xue,et al.  A case study of large-scale parallel I/O analysis and optimization for numerical weather prediction system , 2014, Future Gener. Comput. Syst..

[18]  Charles-André Gandin,et al.  Interaction between single grain solidification and macrosegregation: Application of a cellular automaton—Finite element model , 2007 .

[19]  John Shalf,et al.  Multithreaded global address space communication techniques for Gyrokinetic fusion applications on ultra-scale platforms , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[20]  Lee Margetts,et al.  Parallel finite element analysis using the Intel Xeon Phi , 2016 .

[21]  Fpt Frank Baaijens,et al.  An approach to micro-macro modeling of heterogeneous materials , 2001 .

[22]  Lee Margetts,et al.  The convergence variability of parallel iterative solvers , 2006 .

[23]  Lee Margetts,et al.  Fortran 2008 coarrays , 2015, FORF.

[24]  Matti Ristinmaa,et al.  Multi-scale plasticity modeling: Coupled discrete dislocation and continuum crystal plasticity , 2008 .

[25]  C. Gandin,et al.  Probabilistic modelling of microstructure formation in solidification processes , 1993 .

[26]  Bill Long Additional Parallel Features in Fortran , 2016, FORF.

[27]  Lee Margetts,et al.  Towards mechanism-based simulation of impact damage using exascale computing , 2017 .

[28]  Lee Margetts,et al.  Transient Thermal Finite Element Analysis of CFC-Cu ITER Monoblock Using X-ray Tomography Data , 2015 .

[29]  Ted Belytschko,et al.  Conservation properties of the bridging domain method for coupled molecular/continuum dynamics , 2008 .

[30]  Dierk Raabe,et al.  Interaction between recrystallization and phase transformation during intercritical annealing in a c , 2013 .

[31]  Ronald E. Miller,et al.  Atomistic/continuum coupling in computational materials science , 2003 .

[32]  Ian C. Howard,et al.  Modeling the ductile-brittle transition behavior in thermomechanically controlled rolled steels , 2005 .