Scalable simulation of realistic volume fraction red blood cell flows through vascular networks

High-resolution blood flow simulations have potential for developing better understanding biophysical phenomena at the microscale, such as vasodilation, vasoconstriction and overall vascular resistance. To this end, we present a scalable platform for the simulation of red blood cell (RBC) flows through complex capillaries by modeling the physical system as a viscous fluid with immersed deformable particles. We describe a parallel boundary integral equation solver for general elliptic partial differential equations, which we apply to Stokes flow through blood vessels. We also detail a parallel collision avoiding algorithm to ensure RBCs and the blood vessel remain contact-free. We have scaled our code on Stampede2 at the Texas Advanced Computing Center up to 34,816 cores. Our largest simulation enforces a contact-free state between four billion surface elements and solves for three billion degrees of freedom on one million RBCs and a blood vessel composed from two million patches.

[1]  Ting Ye,et al.  Three-dimensional motion and deformation of a red blood cell in bifurcated microvessels , 2018 .

[2]  Lexing Ying,et al.  A high-order 3D boundary integral equation solver for elliptic PDEs in smooth domains , 2006, J. Comput. Phys..

[3]  Yigang Wang,et al.  DCCD: Distributed N-Body Rigid Continuous Collision Detection for Large-Scale Virtual Environments , 2017 .

[4]  Paris Perdikaris,et al.  An Effective Fractal-Tree Closure Model for Simulating Blood Flow in Large Arterial Networks , 2014, Annals of Biomedical Engineering.

[5]  Ian H. Sloan,et al.  Fully discrete spectral boundary integral methods for Helmholtz problems on smooth closed surfaces in ${\mathbb R}^3$ , 2002, Numerische Mathematik.

[6]  Michael J Shelley,et al.  Computing collision stress in assemblies of active spherocylinders: Applications of a fast and generic geometric method. , 2018, The Journal of chemical physics.

[7]  Gianluca Iaccarino,et al.  Immersed-finite-element method for deformable particle suspensions in viscous and viscoelastic media , 2018, Physical Review E.

[8]  Hong Zhao,et al.  A spectral boundary integral method for flowing blood cells , 2010, J. Comput. Phys..

[9]  John A. Gunnels,et al.  Massively parallel models of the human circulatory system , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[10]  George Biros,et al.  A fast algorithm for simulating vesicle flows in three dimensions , 2011, J. Comput. Phys..

[11]  Michael E. Papka,et al.  A new computational paradigm in multiscale simulations: Application to brain blood flow , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[12]  Eric S. G. Shaqfeh,et al.  Effect of Cytoplasmic Viscosity on Red Blood Cell Migration in Small Arteriole-level Confinements , 2019, bioRxiv.

[13]  Jonathan B. Freund,et al.  Numerical Simulation of Flowing Blood Cells , 2014 .

[14]  Denis Zorin,et al.  Contact-aware simulations of particulate Stokesian suspensions , 2016, J. Comput. Phys..

[15]  Dhairya Malhotra,et al.  Algorithm 967 , 2016, ACM Trans. Math. Softw..

[16]  Hammad Mazhar,et al.  A scalable parallel method for large collision detection problems , 2011 .

[17]  Chwee Teck Lim,et al.  Particle-based simulations of red blood cells-A review. , 2016, Journal of biomechanics.

[18]  Fuchang Liu,et al.  Real-time collision culling of a million bodies on graphics processing units , 2010, SIGGRAPH 2010.

[19]  George Biros,et al.  A numerical method for simulating the dynamics of 3D axisymmetric vesicles suspended in viscous flows , 2009, J. Comput. Phys..

[20]  George Biros,et al.  Boundary integral method for the flow of vesicles with viscosity contrast in three dimensions , 2015, J. Comput. Phys..

[21]  S. Bhattacharjee,et al.  Motion of a spherical particle in a cylindrical channel using arbitrary Lagrangian-Eulerian method. , 2008, Journal of colloid and interface science.

[22]  HarmonDavid,et al.  Interference-aware geometric modeling , 2011 .

[23]  Anna-Karin Tornberg,et al.  A fast integral equation method for solid particles in viscous flow using quadrature by expansion , 2016, J. Comput. Phys..

[24]  C. Pozrikidis.  Boundary Integral and Singularity Methods for Linearized Viscous Flow: Index , 1992 .

[25]  Chwee Teck Lim,et al.  Hybrid smoothed dissipative particle dynamics and immersed boundary method for simulation of red blood cells in flows. , 2017, Physical review. E.

[26]  S. Fang.  A linearization method for generalized complementarity problems , 1984 .

[27]  G. Karniadakis,et al.  Multiscale modeling and simulation of brain blood flow. , 2016, Physics of fluids.

[28]  Leslie Greengard,et al.  A fast algorithm for particle simulations , 1987 .

[29]  H. Power,et al.  Second kind integral equation formulation of Stokes' flows past a particle of arbitary shape , 1987 .

[30]  Hari Sundar,et al.  HykSort: a new variant of hypercube quicksort on distributed memory architectures , 2013, ICS '13.

[31]  Carsten Burstedde,et al.  p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees , 2011, SIAM J. Sci. Comput..

[32]  Denis Zorin,et al.  A fast platform for simulating semi-flexible fiber suspensions applied to cell mechanics , 2016, J. Comput. Phys..

[33]  Dhairya Malhotra,et al.  A parallel algorithm for long-timescale simulation of concentrated vesicle suspensions in three dimensions , 2017 .

[34]  G. Biros,et al.  PVFMM: A Parallel Kernel Independent FMM for Particle and Volume Potentials , 2015 .

[35]  Y. Fung,et al.  Mechanics of the Circulation , 2011, Developments in Cardiovascular Medicine.

[36]  Oscar P. Bruno,et al.  A high-order integral solver for scalar problems of diffraction by screens and apertures in three-dimensional space , 2012, J. Comput. Phys..

[37]  Jaehyuk Huh,et al.  HPCCD: Hybrid Parallel Continuous Collision Detection using CPUs and GPUs , 2009, Comput. Graph. Forum.

[38]  Prosenjit Bagchi,et al.  Direct Numerical Simulation of Cellular-Scale Blood Flow in 3D Microvascular Networks. , 2017, Biophysical journal.

[39]  Myriam Peyrounette,et al.  Multiscale modelling of blood flow in cerebral microcirculation: Details at capillary scale control accuracy at the level of the cortex , 2018, PloS one.

[40]  Andreas Klöckner,et al.  Optimization of fast algorithms for global Quadrature by Expansion using target-specific expansions , 2018, J. Comput. Phys..

[41]  Chunning Ji,et al.  Large scale simulation of red blood cell aggregation in shear flows. , 2013, Journal of biomechanics.

[42]  Leslie Greengard,et al.  Quadrature by expansion: A new method for the evaluation of layer potentials , 2012, J. Comput. Phys..

[43]  Amanda Randles,et al.  A Computational Framework to Assess the Influence of Changes in Vascular Geometry on Blood Flow , 2017, PASC.

[44]  Jonathan B. Freund,et al.  Simulation of Platelet, Thrombus and Erythrocyte Hydrodynamic Interactions in a 3D Arteriole with In Vivo Comparison , 2013, PloS one.

[45]  Prosenjit Bagchi,et al.  A computational approach to modeling cellular-scale blood flow in complex geometry , 2017, J. Comput. Phys..

[46]  William Gropp,et al.  PETSc Users Manual Revision 3.4 , 2016 .

[47]  W. Helfrich.  Elastic Properties of Lipid Bilayers: Theory and Possible Experiments , 1973, Zeitschrift fur Naturforschung. Teil C: Biochemie, Biophysik, Biologie, Virologie.

[48]  Ulrich Rüde,et al.  A Parallel Rigid Body Dynamics Algorithm , 2009, Euro-Par.

[49]  Richard W. Vuduc,et al.  Petascale Direct Numerical Simulation of Blood Flow on 200K Cores and Heterogeneous Architectures , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[50]  Anna-Karin Tornberg,et al.  A 3D boundary integral method for the electrohydrodynamics of surfactant-covered drops , 2018, J. Comput. Phys..

[51]  Anna-Karin Tornberg,et al.  A highly accurate boundary integral equation method for surfactant-laden drops in 3D , 2017, J. Comput. Phys..

[52]  Wolfgang Straßer,et al.  Fast and Scalable CPU/GPU Collision Detection for Rigid and Deformable Surfaces , 2010, Comput. Graph. Forum.

[53]  Eric S. G. Shaqfeh,et al.  Simulation of Red Blood Cell Migration in Small Arterioles: Effect of Cytoplasmic Viscosity , 2019 .

[54]  Andreas Klöckner,et al.  A fast algorithm with error bounds for Quadrature by Expansion , 2018, J. Comput. Phys..

[55]  D. Zorin,et al.  Parallel contact-aware simulations of deformable particles in 3D Stokes flow , 2018, 1812.04719.

[56]  Henny H. Billett,et al.  Hemoglobin and Hematocrit , 1990 .

[57]  Olga Sorkine-Hornung,et al.  Interference-aware geometric modeling , 2011, ACM Trans. Graph..

[58]  Massimo Bernaschi,et al.  The in-silico lab-on-a-chip: petascale and high-throughput simulations of microfluidics at cell resolution , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[59]  P. Canham.  The minimum energy of bending as a possible explanation of the biconcave shape of the human red blood cell. , 1970, Journal of theoretical biology.