Iterative load-balancing method with multigrid level relaxation for particle simulation with short-range interactions

Abstract We developed dynamic load-balancing algorithms for Particle Simulation Methods (PSM) involving short-range interactions, such as Smoothed Particle Hydrodynamics (SPH), Moving Particle Semi-implicit method (MPS), and Discrete Element method (DEM). These are needed to handle billions of particles modeled in large distributed-memory computer systems. Our method utilizes flexible orthogonal domain decomposition, allowing the sub-domain boundaries in the column to be different for each row. The imbalances in the execution time between parallel logical processes are treated as a nonlinear residual. Load-balancing is achieved by minimizing the residual within the framework of an iterative nonlinear solver, combined with a multigrid technique in the local smoother. Our iterative method is suitable for adjusting the sub-domain frequently by monitoring the performance of each computational process because it is computationally cheaper in terms of communication and memory costs than non-iterative methods. Numerical tests demonstrated the ability of our approach to handle workload imbalances arising from a non-uniform particle distribution, differences in particle types, or heterogeneous computer architecture which was difficult with previously proposed methods. We analyzed the parallel efficiency and scalability of our method using Earth simulator and K-computer supercomputer systems.

[1]  Shinji Shimojo,et al.  A new dynamical domain decomposition method for parallel molecular dynamics simulation , 2004, CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005..

[2]  Satoshi Tanaka,et al.  Development of Hierarchical Domain Decomposition Explicit MPS Method and Application to Large-scale Tsunami Analysis with Floating Objects , 2014 .

[3]  Paul W. Cleary,et al.  Simulation of particle flows and breakage in crushers using DEM: Part 1 – Compression crushers , 2015 .

[4]  Corrado Altomare,et al.  Applicability of Smoothed Particle Hydrodynamics for estimation of sea wave impact on coastal structures , 2015 .

[5]  Liang Wang,et al.  The Plasma Simulation Code: A modern particle-in-cell code with patch-based load-balancing , 2016, J. Comput. Phys..

[6]  Mikito Furuichi,et al.  Development of a Stokes flow solver robust to large viscosity jumps using a Schur complement approach with mixed precision arithmetic , 2011, J. Comput. Phys..

[7]  Daisuke Nishiura,et al.  Parallel-vector algorithms for particle simulations on shared-memory multiprocessors , 2011, J. Comput. Phys..

[8]  Daisuke Nishiura,et al.  Design of Nanoparticle Dispersion Process in Stirred Media Mill Using DEM–LES Coupling Method , 2012 .

[9]  Mikito Furuichi,et al.  Implicit solution of the material transport in Stokes flow simulation: Toward thermal convection simulation surrounded by free surface , 2015, Comput. Phys. Commun..

[10]  Moncho Gómez-Gesteira,et al.  New multi-GPU implementation for smoothed particle hydrodynamics on heterogeneous clusters , 2013, Comput. Phys. Commun..

[11]  Mikito Furuichi,et al.  Computational performance of a smoothed particle hydrodynamics simulation for shared-memory parallel computing , 2015, Comput. Phys. Commun..

[12]  Mostafa Safdari Shadloo,et al.  Smoothed particle hydrodynamics method for fluid flows, towards industrial applications: Motivations, current state, and challenges , 2016 .

[13]  Rui M. L. Ferreira,et al.  SPH-DCDEM model for arbitrary geometries in free surface solid-fluid flows , 2016, Comput. Phys. Commun..

[14]  Cees Hesp,et al.  Incidence of HIV in Windhoek, Namibia: Demographic and Socio-Economic Associations , 2011, PloS one.

[15]  Steven J. Plimpton,et al.  A load-balancing algorithm for a parallel electromagnetic particle-in-cell code , 2003 .

[16]  Hiroshi Nakashima,et al.  OhHelp: a scalable domain-decomposing dynamic load balancing for particle-in-cell simulations , 2009, ICS.

[17]  P. Cundall,et al.  A discrete numerical model for granular assemblies , 1979 .

[18]  P. Eberhard,et al.  Parallel load‐balanced simulation for short‐range interaction particle methods with hierarchical particle grouping based on orthogonal recursive bisection , 2008 .

[19]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[20]  D. Bhattacharjee,et al.  Three-dimensional geometry of thrust surfaces and the origin of sinuous thrust traces in orogenic belts: Insights from scaled sandbox experiments , 2014 .

[21]  Srinivas Aluru,et al.  Parallel domain decomposition and load balancing using space-filling curves , 1997, Proceedings Fourth International Conference on High-Performance Computing.

[22]  Carsten Kutzner,et al.  GROMACS 4:  Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation. , 2008, Journal of chemical theory and computation.

[23]  Kohei Murotani,et al.  Performance improvements of differential operators code for MPS method on GPU , 2015 .

[24]  B. Rogers,et al.  GPUs, a New Tool of Acceleration in CFD: Efficiency and Reliability on Smoothed Particle Hydrodynamics Methods , 2011, PloS one.

[25]  Hitoshi Gotoh,et al.  GPU-acceleration for Moving Particle Semi-Implicit method , 2011 .

[26]  Moncho Gómez-Gesteira,et al.  Optimization strategies for CPU and GPU implementations of a smoothed particle hydrodynamics method , 2013, Comput. Phys. Commun..

[27]  S. Koshizuka,et al.  Moving-Particle Semi-Implicit Method for Fragmentation of Incompressible Fluid , 1996 .

[28]  D. Keyes,et al.  Jacobian-free Newton-Krylov methods: a survey of approaches and applications , 2004 .

[29]  Stefan Heinrich,et al.  Modeling of the Spray Zone for Particle Wetting in a Fluidized Bed , 2013 .

[30]  Magnus Evertsson,et al.  The contribution of DEM to the science of comminution , 2013 .

[31]  Nobuyasu Ito,et al.  Huge-scale molecular dynamics simulation of multibubble nuclei , 2013, Comput. Phys. Commun..

[32]  George W. Bergantz,et al.  Open-system dynamics and mixing in magma mushes , 2015 .

[33]  Paul W. Cleary,et al.  Modelling confined multi-material heat and mass flows using SPH , 1998 .

[34]  James N. Glosli,et al.  Dynamic load balancing algorithm for molecular dynamics based on Voronoi cells domain decompositions , 2012, Comput. Phys. Commun..

[35]  J. Monaghan.  Smoothed particle hydrodynamics , 2005 .

[36]  Wolfgang Meyer,et al.  Load balancing algorithms based on gradient methods and their analysis through algebraic graph theory , 2008, J. Parallel Distributed Comput..

[37]  Robert A. Dalrymple,et al.  SPHysics - development of a free-surface fluid solver - Part 2: Efficiency and test cases , 2012, Comput. Geosci..

[38]  Ahmed F. Ghoniem,et al.  K-means clustering for optimal partitioning and dynamic load balancing of parallel hierarchical N-body simulations , 2005 .