Efficient parallel CFD-DEM simulations using OpenMP

The paper describes parallelization strategies for the Discrete Element Method (DEM) used for simulating dense particulate systems coupled to Computational Fluid Dynamics (CFD). While the field equations of CFD are best parallelized by spatial domain decomposition techniques, the N-body particulate phase is best parallelized over the number of particles. When the two are coupled together, both modes are needed for efficient parallelization. It is shown that under these requirements, OpenMP thread based parallelization has advantages over MPI processes. Two representative examples, fairly typical of dense fluid-particulate systems are investigated, including the validation of the DEM-CFD and thermal-DEM implementation with experiments. Fluidized bed calculations are performed on beds with uniform particle loading, parallelized with MPI and OpenMP. It is shown that as the number of processing cores and the number of particles increase, the communication overhead of building ghost particle lists at processor boundaries dominates time to solution, and OpenMP which does not require this step is about twice as fast as MPI. In rotary kiln heat transfer calculations, which are characterized by spatially non-uniform particle distributions, the low overhead of switching the parallelization mode in OpenMP eliminates the load imbalances, but introduces increased overheads in fetching non-local data. In spite of this, it is shown that OpenMP is between 50-90% faster than MPI. OpenMP parallelization of a CFD-DEM code.Validation of the CFD-DEM code.OpenMP has clear advantage over MPI for dynamic load imbalance case of fluid-particulate system.Parallel performance comparison using fluidized bed and rotary kiln simulations.Full scale rotary kiln simulation using OpenMP parallel code.

[1]  Danesh K. Tafti,et al.  Particle Scale Heat Transfer Analysis in Rotary Kiln , 2012 .

[2]  Rui Liu,et al.  OpenMP parallelism for fluid and fluid-particulate systems , 2012, Parallel Comput..

[3]  Ivo F. Sbalzarini,et al.  Large‐scale parallel discrete element simulations of granular flow , 2009 .

[4]  Christopher E. Brennen,et al.  Computer simulation of granular shear flows , 1985, Journal of Fluid Mechanics.

[5]  Lynn F. Gladden,et al.  Validation of a discrete element model using magnetic resonance measurements , 2009 .

[6]  Keizo Yabumoto,et al.  Spontaneous structures in three-dimensional bubbling gas-fluidized bed by parallel DEM–CFD coupling simulation , 2008 .

[7]  T. B. Anderson,et al.  Fluid Mechanical Description of Fluidized Beds. Equations of Motion , 1967 .

[8]  Joseph J. McCarthy,et al.  Heat conduction in granular materials , 2001 .

[9]  P. Cundall,et al.  A discrete numerical model for granular assemblies , 1979 .

[10]  Naresh K. C. Selvarasu,et al.  Hydrodynamic effects of compliance mismatch in stented arteries. , 2011, Journal of biomechanical engineering.

[11]  M. M. Chen,et al.  A theoretical analysis of heat transfer due to particle impact , 1988 .

[12]  G. Batchelor,et al.  Thermal or electrical conduction through a granular material , 1977, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences.

[13]  S. Balachandar,et al.  Computations of flow and heat transfer in parallel-plate fin heat exchangers on the CM-5: effects of flow unsteadiness and three-dimensionality , 1997 .

[14]  J. Kuipers,et al.  Discrete particle simulation of bubble and slug formation in a two-dimensional gas-fluidised bed: A hard-sphere approach. , 1996 .

[15]  Gilles Flamant,et al.  DEM-LES simulation of coal combustion in a bubbling fluidized bed Part II: coal combustion at the particle level , 2004 .

[16]  Ivo F. Sbalzarini,et al.  PPM - A highly efficient parallel particle-mesh library for the simulation of continuum systems , 2006, J. Comput. Phys..

[17]  Jérôme Fortin,et al.  Discrete modeling of granular flow with thermal transfer: Application to the discharge of silos , 2009 .

[18]  Danesh K. Tafti,et al.  A Parallel Computing Framework for Dynamic Power Balancing in Adaptive Mesh Refinement Applications , 2000 .

[19]  D. Gunn.  Transfer of heat or mass to particles in fixed and fluidised beds , 1978 .

[20]  Algirdas Maknickas,et al.  Parallel DEM Software for Simulation of Granular Media , 2006, Informatica.

[21]  Danesh K. Tafti,et al.  Modeling of soot deposition in wavy-fin exhaust gas recirculator coolers , 2011 .

[22]  Y. Tsuji,et al.  Discrete particle simulation of two-dimensional fluidized bed , 1993 .

[23]  C. Wen.  Mechanics of Fluidization , 1966 .

[24]  Hans-Peter Kersken,et al.  HICFD: Highly Efficient Implementation of CFD Codes for HPC Many-Core Architectures , 2010, CHPC.

[25]  Danesh K. Tafti,et al.  Time-accurate Techniques For Turbulent HeatTransfer Analysis In Complex Geometries , 2010 .

[26]  Jay N. Meegoda,et al.  Micro‐mechanical simulation of geotechnical problems using massively parallel computers , 2003 .

[27]  Fernando J. Muzzio,et al.  Experimentally validated computations of heat transfer in granular materials in rotary calciners , 2010 .

[28]  Colin Thornton,et al.  Parallelization of a Lagrangian–Eulerian DEM/CFD code for application to fluidized beds , 2011 .

[29]  Algirdas Maknickas,et al.  Parallel discrete element simulation of poly-dispersed granular material , 2010, Adv. Eng. Softw..

[30]  Danesh K. Tafti,et al.  Effect of Wing Flexibility on Lift and Thrust Production in Flapping Flight , 2010 .

[31]  Olivier Desjardins,et al.  An Euler-Lagrange strategy for simulating particle-laden flows , 2013, J. Comput. Phys..

[32]  Danesh K. Tafti,et al.  Development of parallel DEM for the open source code MFIX , 2013 .

[33]  Dimitri J. Mavriplis,et al.  Parallel Performance Investigations of an Unstructured Mesh Navier-Stokes Solver , 2000, Int. J. High Perform. Comput. Appl..

[34]  Jun Wang,et al.  Parallel CFD-DEM for Fluid-Particle Systems , 2004 .

[35]  Joseph J. McCarthy,et al.  Stress effects on the conductivity of particulate beds , 2002 .

[36]  Steve Plimpton,et al.  Fast parallel algorithms for short-range molecular dynamics , 1993 .

[37]  Niels G. Deen,et al.  Parallelization of an Euler-Lagrange model using mixed domain decomposition and a mirror domain technique: Application to dispersed gas-liquid two-phase flow , 2006, J. Comput. Phys..

[38]  Asensio Oliva Llena,et al.  Parallelization of a DEM/CFD code for the numerical simulation of particle-laden turbulent flows , 2011 .

[39]  Mikio Sakai,et al.  Parallel computing of discrete element method on multi-core processors , 2011 .

[40]  S. Ergun.  Fluid flow through packed columns , 1952 .

[41]  Sergei Stepanovich Zabrodskii.  Hydrodynamics and heat transfer in fluidized beds , 1966 .

[42]  T. Rung,et al.  Hybrid MPI/OpenMP parallelization of an Euler–Lagrange approach to cavitation modelling , 2013 .

[43]  Rainald Löhner,et al.  Timings of an Unstructured-Grid CFD Code on Common Hardware Platforms and Compilers , 2007 .

[44]  Frank P. Incropera,et al.  Fundamentals of Heat and Mass Transfer , 1981 .

[45]  K. Malone,et al.  Particle-scale simulation of heat transfer in liquid-fluidised beds , 2008 .

[46]  Danesh K. Tafti,et al.  A Parallel Adaptive Mesh Refinement Algorithm for Solving Nonlinear Dynamical Systems , 2004, Int. J. High Perform. Comput. Appl..