Space decomposition based parallelization solutions for the combined finite–discrete element method in 2D

The combined finite–discrete element method (FDEM) belongs to a family of methods of computational mechanics of discontinua. The method is suitable for problems of discontinua, where particles are deformable and can fracture or fragment. The applications of FDEM have spread over a number of disciplines including rock mechanics, where problems like mining, mineral processing or rock blasting can be solved by employing FDEM. In this work, a novel approach for the parallelization of two-dimensional (2D) FDEM aiming at clusters and desktop computers is developed. Dynamic domain decomposition based parallelization solvers covering all aspects of FDEM have been developed. These have been implemented into the open source Y2D software package and have been tested on a PC cluster. The overall performance and scalability of the parallel code have been studied using numerical examples. The results obtained confirm the suitability of the parallel implementation for solving large scale problems.

[1]  B. Mohanty,et al.  Fracture Toughness Measurements and Acoustic Emission Activity in Brittle Rocks , 2006 .

[2]  D. C. Rapaport,et al.  Multibillion-atom molecular dynamics simulation: Design considerations for vector-parallel processing , 2006, Comput. Phys. Commun..

[3]  K. Esselink,et al.  Efficient Parallel Implementation of Molecular Dynamics on a Toroidal Network. Part II. Multi-particle Potentials , 1993 .

[4]  M. Shephard,et al.  Load balancing for the parallel adaptive solution of partial differential equations , 1994 .

[5]  Mark T. Jones,et al.  Computational Results for Parallel Unstructured Mesh Computations , 1994 .

[6]  Karen Dragon Devine,et al.  Parallel adaptive hp -refinement techniques for conservation laws , 1996 .

[7]  Antonio Munjiza,et al.  A framework for grand scale parallelization of the combined finite discrete element method in 2d , 2014, CPM 2014.

[8]  M. Pernice,et al.  PVM: Parallel Virtual Machine - A User's Guide and Tutorial for Networked Parallel Computing [Book Review] , 1996, IEEE Parallel & Distributed Technology: Systems & Applications.

[9]  Peng Wang,et al.  Efficient GPU-accelerated molecular dynamics simulation of solid covalent crystals , 2013, Comput. Phys. Commun..

[10]  Antonio Munjiza,et al.  Validation of a three-dimensional Finite-Discrete Element Method using experimental results of the Split Hopkinson Pressure Bar test , 2014 .

[11]  Karsten Schwan,et al.  Design and Analysis of a Parallel Molecular Dynamics Application , 1996, J. Parallel Distributed Comput..

[12]  Ananth Grama,et al.  PuReMD-GPU: A reactive molecular dynamics simulation package for GPUs , 2014, J. Comput. Phys..

[13]  Steven J. Plimpton,et al.  Scalable parallel molecular dynamics on MIMD supercomputers , 1992, Proceedings Scalable High Performance Computing Conference SHPCC-92..

[14]  Keizo Yabumoto,et al.  Spontaneous structures in three-dimensional bubbling gas-fluidized bed by parallel DEM–CFD coupling simulation , 2008 .

[15]  S. Succi.  The Lattice Boltzmann Equation for Fluid Dynamics and Beyond , 2001 .

[16]  Christine M. Hrenya,et al.  Challenges of DEM: I. Competing bottlenecks in parallelization of gas–solid flows , 2014 .

[17]  Message Passing Interface Forum.  MPI: A message - passing interface standard , 1994 .

[18]  Daichao Sheng,et al.  A saturated discrete particle model and characteristic‐based SPH method in granular materials , 2007 .

[19]  Y.-F. Deng,et al.  Molecular dynamics on distributed-memory MIMD computers with load balancing , 1994 .

[20]  Mikio Sakai,et al.  Parallel computing of discrete element method on multi-core processors , 2011 .

[21]  John-Paul Latham,et al.  Detonation gas model for combined finite‐discrete element simulation of fracture and fragmentation , 2000 .

[22]  Daisuke Nishiura,et al.  Parallel-vector algorithms for particle simulations on shared-memory multiprocessors , 2011, J. Comput. Phys..

[23]  A. Munjiza,et al.  NBS contact detection algorithm for bodies of similar size , 1998 .

[24]  S. Chaplot,et al.  Parallelization in classical molecular dynamics simulation and applications , 2006 .

[25]  M. Everett,et al.  Parallel dynamic graph-partitioning for unstructured meshes , 1997 .

[26]  David Kaeli,et al.  Introduction to Parallel Programming , 2013 .

[27]  Rajiv K. Kalia,et al.  Parallel multiple-time-step molecular dynamics with three-body interaction , 1993 .

[28]  Benedict D. Rogers,et al.  Towards accelerating smoothed particle hydrodynamics simulations for free-surface flows on multi-GPU clusters , 2012, J. Parallel Distributed Comput..

[29]  L. R. Scott,et al.  Parallelizing molecular dynamics using spatial decomposition , 1994, Proceedings of IEEE Scalable High Performance Computing Conference.

[30]  Y. Feng,et al.  Coupled lattice Boltzmann method and discrete element modelling of particle transport in turbulent fluid flows: Computational issues , 2007 .

[31]  A. Munjiza.  The Combined Finite-Discrete Element Method , 2004 .

[32]  Steven F. Quigley,et al.  A fast scalable implementation of the two-dimensional triangular Discrete Element Method on a GPU platform , 2013, Adv. Eng. Softw..

[33]  Hans-Wolfgang Loidl,et al.  A Survey of High-Level Parallel Programming Models , 2013 .

[34]  Jong-Shinn Wu,et al.  Parallel implementation of molecular dynamics simulation for short-ranged interaction , 2005, Comput. Phys. Commun..

[35]  Z. C.,et al.  Analysis of The Generalized Dimension Exchange Method forDynamic Load Balancing , 1992 .

[36]  William Smith,et al.  Molecular dynamics on hypercube parallel computers , 1991 .

[37]  Jie Cheng,et al.  Programming Massively Parallel Processors. A Hands-on Approach , 2010, Scalable Comput. Pract. Exp..

[38]  G. Horton.  A Multi-Level Diffusion Method for Dynamic Load Balancing , 1993, Parallel Comput..

[39]  Tiberiu Rotaru,et al.  Dynamic load balancing by diffusion in heterogeneous systems , 2004, J. Parallel Distributed Comput..

[40]  Shahid H. Bokhari,et al.  A Partitioning Strategy for Nonuniform Problems on Multiprocessors , 1987, IEEE Transactions on Computers.

[41]  Daisuke Nishiura,et al.  ppohDEM: Computational performance for open source code of the discrete element method , 2014, Comput. Phys. Commun..

[42]  Gary L. Miller,et al.  A unified geometric approach to graph separators , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[43]  James Andrew McCammon,et al.  Parallelization of a molecular dynamics non-bonded force algorithm for MIMD architecture , 1990, Comput. Chem..

[44]  Ted Belytschko,et al.  Object-oriented parallelization of explicit structural dynamics with PVM , 1998 .

[45]  Ted Belytschko,et al.  Parallel processing for a discrete element program , 1999 .

[46]  P. Cundall.  A computer model for simulating progressive, large-scale movements in blocky rock systems , 1971 .

[47]  Luc T. Wille,et al.  Parallel molecular dynamics simulations for short-ranged many-body potentials☆ , 2000 .

[48]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[49]  David R. Owen,et al.  Parallelised finite/discrete element simulation of multi‐fracturing solids and discrete systems , 2001 .

[50]  D. V. Griffiths,et al.  Programming the finite element method , 1982 .

[51]  George Karypis,et al.  Parmetis parallel graph partitioning and sparse matrix ordering library , 1997 .

[52]  Horst D. Simon,et al.  Partitioning of unstructured problems for parallel processing , 1991 .

[53]  Ananth Grama,et al.  Parallel reactive molecular dynamics: Numerical methods and algorithmic techniques , 2012, Parallel Comput..

[54]  Alex Pothen,et al.  PARTITIONING SPARSE MATRICES WITH EIGENVECTORS OF GRAPHS* , 1990 .

[55]  Jingwei Zheng,et al.  GPU-based parallel algorithm for particle contact detection and its application in self-compacting concrete flow simulations , 2012 .

[56]  Barbara Chapman,et al.  Using OpenMP - portable shared memory parallel programming , 2007, Scientific and engineering computation.

[57]  Valerie E. Taylor,et al.  Parallel Molecular Dynamics: Implications for Massively Parallel Machines , 1997, J. Parallel Distributed Comput..

[58]  Esteban Rougier,et al.  MR linear contact detection algorithm , 2006 .

[59]  Esteban Rougier,et al.  Computational Mechanics of Discontinua , 2011 .

[60]  Danesh K. Tafti,et al.  Hybrid parallelism in MFIX CFD-DEM using OpenMP , 2014 .

[61]  Chunning Ji,et al.  Large scale simulation of red blood cell aggregation in shear flows. , 2013, Journal of biomechanics.

[62]  J. E. Boillat,et al.  A dynamic load-balancing algorithm for molecular dynamics simulation on multi-processor systems , 1991 .

[63]  Colin Thornton,et al.  Parallelization of a Lagrangian–Eulerian DEM/CFD code for application to fluidized beds , 2011 .

[64]  D. C. Rapaport,et al.  The Art of Molecular Dynamics Simulation , 1997 .

[65]  P. Cundall,et al.  A discrete numerical model for granular assemblies , 1979 .

[66]  Damien Violeau,et al.  Numerical modelling of complex turbulent free‐surface flows with the SPH method: an overview , 2007 .

[67]  Robert Michael Kirby,et al.  Parallel Scientific Computing in C++ and MPI - A Seamless Approach to Parallel Algorithms and their Implementation , 2003 .

[68]  Göran Frenning,et al.  An efficient finite/discrete element procedure for simulating compression of 3D particle assemblies , 2008 .

[69]  D. C. Rapaport,et al.  Multi-million particle molecular dynamics I. Design considerations for vector processing , 1991 .

[70]  J. Tinsley Oden,et al.  Problem decomposition for adaptive hp finite element methods , 1995 .

[71]  Kwang Jin Oh,et al.  A general purpose parallel molecular dynamics simulation program , 2006, Comput. Phys. Commun..

[72]  Benjamin J. Glasser,et al.  Large-scale powder mixer simulations using massively parallel GPUarchitectures , 2010 .

[73]  D. C. Rapaport,et al.  Multi-million particle molecular dynamics: II. Design considerations for distributed processing , 1991 .

[74]  Wei Ge,et al.  GPU-based discrete element simulation on a tote blender for performance improvement , 2013 .

[75]  Jian Yang,et al.  Numerical Analysis of Cold-formed sigms steel beams , 2010 .

[76]  K. Apostolou,et al.  Discrete element simulation of liquid-particle flows , 2008, Comput. Chem. Eng..

[77]  George Cybenko,et al.  Dynamic Load Balancing for Distributed Memory Multiprocessors , 1989, J. Parallel Distributed Comput..

[78]  F. J. WANG,et al.  Parallelisation for finite-discrete element analysis in a distributed-memory environment , 2004, Int. J. Comput. Eng. Sci..

[79]  Moncho Gómez-Gesteira,et al.  New multi-GPU implementation for smoothed particle hydrodynamics on heterogeneous clusters , 2013, Comput. Phys. Commun..

[80]  Gregory Karagiorgos,et al.  Accelerated diffusion algorithms for dynamic load balancing , 2002, Inf. Process. Lett..

[81]  Florian Müller-Plathe,et al.  Parallelising a molecular dynamics algorithm on a multi-processor workstation , 1990 .

[82]  Y. Feng,et al.  Coupled lattice Boltzmann and discrete element modelling of fluid-particle interaction problems , 2007 .

[83]  Richard B. P. Burrows.  Dynamic load balancing , 1997 .

[84]  Algirdas Maknickas,et al.  Parallel discrete element simulation of poly-dispersed granular material , 2010, Adv. Eng. Softw..

[85]  Michael J. Flynn,et al.  Very high-speed computing systems , 1966 .

[86]  Laxmikant V. Kalé,et al.  NAMD: a Parallel, Object-Oriented Molecular Dynamics Program , 1996, Int. J. High Perform. Comput. Appl..

[87]  Antonio Munjiza,et al.  Combined single and smeared crack model in combined finite-discrete element analysis , 1999 .

[88]  David A. Patterson,et al.  Computer Organization and Design: The Hardware Software Interface ARM Edition , 2016 .

[89]  Matthias Durr,et al.  Molecular Dynamics Simulation Elementary Methods , 2016 .

[90]  Andreas W. Götz,et al.  SPFP: Speed without compromise - A mixed precision model for GPU accelerated molecular dynamics simulations , 2013, Comput. Phys. Commun..

[91]  Y. Kallinderisy,et al.  Parallel Load Balancing for Dynamic Execution Environments , 1996 .

[92]  J. Valverde.  Molecular Modelling: Principles and Applications , 2001 .

[93]  Richard J. Sadus,et al.  Molecular Simulation of Fluids: Theory, Algorithms and Object-Orientation , 1999 .

[94]  Bruce Hendrickson,et al.  A Multi-Level Algorithm For Partitioning Graphs , 1995, Proceedings of the IEEE/ACM SC95 Conference.

[95]  Y. F. Hu,et al.  An improved diffusion algorithm for dynamic load balancing , 1999, Parallel Comput..

[96]  Nikos Chrisochoides,et al.  MULTITHREADED MODEL FOR DYNAMIC LOAD BALANCING PARALLEL ADAPTIVE PDE COMPUTATIONS , 1995 .

[97]  D. C. Rapaport,et al.  Multi-million particle molecular dynamics: III. Design considerations for data-parallel processing , 1993 .

[98]  Jianhui Li,et al.  Parallel algorithms for molecular dynamics with induction forces , 2008, Comput. Phys. Commun..

[99]  Lei Zhang,et al.  A GPU-Based Parallel Procedure for Nonlinear Analysis of Complex Structures Using a Coupled FEM/DEM Approach , 2013 .

[100]  Guirong Liu,et al.  Smoothed Particle Hydrodynamics: A Meshfree Particle Method , 2003 .

[101]  Paul W. Cleary,et al.  A parallel discrete element method for industrial granular flow simulations , 1999 .

[102]  O. K. Mahabadi,et al.  Y-Geo: New Combined Finite-Discrete Element Numerical Code for Geomechanical Applications , 2012 .

[103]  John D. Owens,et al.  Efficient Parallel Scan Algorithms for Manycore GPUs , 2010, Scientific Computing with Multicore and Accelerators.

[104]  Paul W. Cleary,et al.  DEM modelling of industrial granular flows: 3D case studies and the effect of particle shape on hopper discharge , 2002 .

[105]  Giovanni Grasselli,et al.  Continuum–discontinuum analysis of failure mechanisms around unsupported circular excavations in anisotropic clay shales , 2014 .

[106]  Javier Bonet,et al.  Dynamic refinement and boundary contact forces in SPH with applications in fluid flow problems , 2007 .

[107]  Vijay S. Pande,et al.  Accelerating molecular dynamic simulation on graphics processing units , 2009, J. Comput. Chem..

[108]  Sergio Sandoval Reyes.  Parallel Programing with MPI , 2009 .

[109]  Kwang Jin Oh,et al.  A parallel molecular dynamics simulation scheme for a molecular system with bond constraints in NPT ensemble , 2006, Comput. Phys. Commun..

[110]  Y. T. Feng,et al.  DYNAMIC DOMAIN DECOMPOSITION AND LOAD BALANCING IN PARALLEL SIMULATION OF FINITE/DISCRETE ELEMENTS , 2000 .

[111]  Wei Ge,et al.  Parallelization of pseudo-particle modeling and its application in simulating gas-solid fluidization , 2009 .

[112]  Peter Sanders.  Analysis of nearest neighbor load balancing algorithms for random loads , 1999, Parallel Comput..

[113]  Robert B. Ross,et al.  Using MPI-2: Advanced Features of the Message Passing Interface , 2003, CLUSTER.

[114]  Michael Allen,et al.  Parallel programming: techniques and applications using networked workstations and parallel computers , 1998 .

[115]  D. R. Emerson,et al.  An optimal migration algorithm for dynamic load balancing , 1998, Concurr. Pract. Exp..

[116]  Rohit Chandra,et al.  Parallel programming in openMP , 2000 .

[117]  G. Bird.  Molecular Gas Dynamics and the Direct Simulation of Gas Flows , 1994 .

[118]  Zhi Zong,et al.  Smoothed particle hydrodynamics for numerical simulation of underwater explosion , 2003 .

[119]  Martin G. Everett,et al.  Parallel Dynamic Graph Partitioning for Adaptive Unstructured Meshes , 1997, J. Parallel Distributed Comput..

[120]  Antonio Munjiza,et al.  Penalty function method for combined finite–discrete element systems comprising large number of separate bodies , 2000 .

[121]  Long Chen.  FINITE ELEMENT METHOD , 2013 .

[122]  Martin G. Everett,et al.  A Localized Algorithm for Optimizing Unstructured Mesh Partitions , 1995, Int. J. High Perform. Comput. Appl..

[123]  Daniel N. Wilke,et al.  Development of a convex polyhedral discrete element simulation framework for NVIDIA Kepler based GPUs , 2014, J. Comput. Appl. Math..

[124]  James R. Larus,et al.  Software and the Concurrency Revolution , 2005, ACM Queue.

[125]  O. K. Mahabadi,et al.  Investigating the Influence of Micro-scale Heterogeneity and Microstructure on the Failure and Mechanical Behaviour of Geomaterials , 2012 .

[126]  M. Willebeek-LeMair,et al.  A localized dynamic load balancing strategy for highly parallel systems , 1990, [1990 Proceedings] The Third Symposium on the Frontiers of Massively Parallel Computation.

[127]  John Zahorjan,et al.  Dynamic-domain-decomposition parallel molecular dynamics , 1997 .

[128]  O. K. Mahabadi,et al.  Y-GUI: A graphical user interface and pre-processor for the combined finite-discrete element code, Y2D, incorporating material heterogeneity , 2010, Comput. Geosci..

[129]  Rajkumar Buyya.  Architectures and systems , 1999 .

[130]  Peter S. Pacheco.  Parallel programming with MPI , 1996 .

[131]  Anthony P. Reeves,et al.  Strategies for Dynamic Load Balancing on Highly Parallel Computers , 1993, IEEE Trans. Parallel Distributed Syst..

[132]  Schiava D’Albano,et al.  Computational and algorithmic solutions for large scale combined finite-discrete elements simulations , 2014 .

[133]  Brian W. Kernighan,et al.  An efficient heuristic procedure for partitioning graphs , 1970, Bell Syst. Tech. J..

[134]  Francis C. M. Lau,et al.  Decentralized Remapping of Data Parallel Applications in Distributed Memory Multiprocessors , 1997, Concurr. Pract. Exp..

[135]  Vishal Mehra,et al.  High velocity impact of metal sphere on thin metallic plates: A comparative smooth particle hydrodynamics study , 2006, J. Comput. Phys..

[136]  Theodoros E. Karakasidis,et al.  Parallel short range molecular dynamics simulations on computer clusters: Performance evaluation and modeling , 2005, Math. Comput. Model..

[137]  Marc Rieffel,et al.  A Load Balancing Technique for Multiphase Computations , 1997 .

[138]  Wei Ge,et al.  Parallel implementation of macro-scale pseudo-particle simulation for particle-fluid systems , 2005, Comput. Chem. Eng..

[139]  R. Bisseling,et al.  Partitioning 3D space for parallel many-particle simulations , 2001, cond-mat/0105272.

[140]  Roy D. Williams,et al.  Performance of dynamic load balancing algorithms for unstructured mesh calculations , 1991, Concurr. Pract. Exp..

[141]  D. Owen,et al.  A combined finite‐discrete element method in transient dynamics of fracturing solids , 1995 .

[142]  Yusheng Feng,et al.  Domain Decomposition for Adaptive hp Finite Element Methods , 1994 .

[143]  T. Rabczuk,et al.  Simulation of high velocity concrete fragmentation using SPH/MLSPH , 2003 .

[144]  O. K. Mahabadi,et al.  Influence of pre-existing discontinuities and bedding planes on hydraulic fracturing initiation , 2015 .

[145]  Herb Sutter,et al.  The Free Lunch Is Over A Fundamental Turn Toward Concurrency in Software , 2013 .

[146]  Roger I. Tanner,et al.  A parallel algorithm for molecular dynamics simulation of branched molecules , 2003 .

[147]  Steven J. Plimpton,et al.  Parallel Many-Body Simulations without All-to-All Communication , 1995, J. Parallel Distributed Comput..

[148]  O. K. Mahabadi,et al.  An Example of Realistic Modelling of Rock Dynamics Problems: FEM/DEM Simulation of Dynamic Brazilian Test on Barre Granite , 2010 .

[149]  Bruce Hendrickson,et al.  Dynamic load balancing in computational mechanics , 2000 .

[150]  Roger I. Tanner,et al.  Parallel simulation of shear flow of polymers between structured walls by molecular dynamics simulation on PVM , 1997 .

[151]  Curt Jones,et al.  A Heuristic for Reducing Fill-In in Sparse Matrix Factorization , 1993, PPSC.

[152]  Steve Plimpton,et al.  Fast parallel algorithms for short-range molecular dynamics , 1993 .

[153]  Guirong Liu,et al.  Computer simulation of high explosive explosion using smoothed particle hydrodynamics methodology , 2003 .