Topics in ultrascale scientific computing with application in biomedical modeling

In this Thesis we focus on simulations of blood flow in three-dimensional patient-specific arterial networks. We employ high-order spectral/ hp-element spatial discretization and concentrate on computational efficiency in solving multi-million degrees of freedom (DOF) flow problems on petaflop computers. We develop new two-level domain decomposition method and multilevel communicating interface for ultra-parallel flow simulations. Specifically, at the coarse level the computational domain is subdivided into several big patches. Within each patch a spectral element discretization (fine level) is employed. New interface conditions for the Navier-Stokes equations are developed. The proposed numerical approach has been tested in arterial flow simulations with up to 147 arteries. Solution of 2.87B DOF problem was computed on 18,576 processors in less than one second at each time step. A scalable and fast parallel low-energy bases preconditioner (LEBP) in conjunction with coarse-space linear vertex solver is developed. We provide details on optimization, parallel performance and implementation of the coarse-space solver and show scalability of LEBP on thousands processors of the IBM BlueGene/L and the Cray XT3. An embarrassingly parallel but extremely efficient accelerator for iterative solver has been proposed. The new approach reduces the number of conjugate gradient iterations and exhibits grid independent scaling, while the computational overhead is negligible. A new type of outflow boundary condition for networks with multiple outlets has been developed. The method is based on a time-dependent resistance-capacitance model, where the resistance values are related to the measured flowrates. The aforementioned methods have been employed to study unsteady flow in patient-specific intracranial arterial trees. Results of a comparative study of 3D rigid wall and 1D flexible wall modeling of flow in complex arterial networks are presented. Transient turbulent flow in a carotid arterial bifurcation with a stenosed internal carotid artery has been studied in details. To analyze the intermittent in time and space laminar-turbulent flow a new methodology based on time- and space-window Proper Orthogonal Decomposition (POD) is proposed. A simplified version of the POD analysis that utilizes 2D slices only - more appropriate in the clinical setting - is also investigated.

[1]  K. Parker,et al.  Wave propagation in a model of the arterial circulation. , 2004, Journal of biomechanics.

[2]  George Em Karniadakis,et al.  Spectral Element and hp Methods , 2004 .

[3]  L. Trefethen,et al.  Spectra and Pseudospectra , 2020 .

[4]  C. Bernardi,et al.  Approximations spectrales de problèmes aux limites elliptiques , 2003 .

[5]  Suchuan Dong,et al.  Cross-site computations on the TeraGrid , 2005, Computing in Science & Engineering.

[6]  Jan S. Hesthaven,et al.  From Electrostatics to Almost Optimal Nodal Sets for Polynomial Interpolation in a Simplex , 1998 .

[7]  G. Karniadakis,et al.  Spectral/hp Element Methods for CFD , 1999 .

[8]  R. Shadwick,et al.  CIRCULATORY MECHANICS IN THE TOAD BUFO MARINUS , 2005 .

[9]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[10]  Charles A. Taylor,et al.  Fractal network model for simulating abdominal and lower extremity blood flow during resting and exercise conditions , 2007, Computer methods in biomechanics and biomedical engineering.

[11]  Ivo Babuška,et al.  The optimal symmetrical points for polynomial interpolation of real functions in the tetrahedron , 1995 .

[12]  M. Zamir.  On fractal properties of arterial trees. , 1999, Journal of theoretical biology.

[13]  Mark A. Taylor,et al.  A Cardinal Function Algorithm for Computing Multivariate Quadrature Points , 2007, SIAM J. Numer. Anal..

[14]  George Em Karniadakis,et al.  De-aliasing on non-uniform grids: algorithms and applications , 2003 .

[15]  D. Chungcharoen,et al.  GENESIS OF KOROTKOFF SOUNDS. , 1964, The American journal of physiology.

[16]  W. Welkowitz,et al.  Dynamics of the sounds caused by partially occluded femoral arteries in dogs , 1994, Annals of Biomedical Engineering.

[17]  Pablo J. Blanco,et al.  Multidimensional modelling for the carotid artery blood flow , 2006 .

[18]  P. Fischer,et al.  Direct numerical simulation of stenotic flows. Part 1. Steady flow , 2007, Journal of Fluid Mechanics.

[19]  Olof B. Widlund,et al.  A Polylogarithmic Bound for an Iterative Substructuring Method for Spectral Elements in Three Dimensions , 1996 .

[20]  Alfio Quarteroni,et al.  Numerical Treatment of Defective Boundary Conditions for the Navier-Stokes Equations , 2002, SIAM J. Numer. Anal..

[21]  Francesca Rapetti,et al.  Overlapping Schwarz Methods for Fekete and Gauss-Lobatto Spectral Elements , 2007, SIAM J. Sci. Comput..

[22]  N. Cheshire,et al.  Combined MR imaging and numerical simulation of flow in realistic arterial bypass graft models. , 2002, Biorheology.

[23]  D Saloner,et al.  Experimental flow studies in exact-replica phantoms of atherosclerotic carotid bifurcations under steady input conditions. , 2003, Journal of biomechanical engineering.

[24]  D. F. Young,et al.  Computer simulation of arterial flow with applications to arterial and aortic stenoses. , 1992, Journal of biomechanics.

[25]  Jack Dongarra,et al.  MPI: The Complete Reference , 1996 .

[26]  C. D. Murray.  THE PHYSIOLOGICAL PRINCIPLE OF MINIMUM WORK , 1931, The Journal of general physiology.

[27]  R. A. Bajura,et al.  Transition in a two-dimensional plane wall jet , 1975, Journal of Fluid Mechanics.

[28]  Spencer J. Sherwin,et al.  High-order algorithms for vascular flow modelling , 2002 .

[29]  Paul F. Fischer,et al.  Fast Parallel Direct Solvers for Coarse Grid Problems , 2001, J. Parallel Distributed Comput..

[30]  D. Saloner,et al.  Numerical analysis of flow through a severely stenotic carotid artery bifurcation. , 2002, Journal of biomechanical engineering.

[31]  Guy Marchal,et al.  Nonrigid registration for subtraction CT angiography applied to the carotids and cranial arteries. , 2007, Academic radiology.

[32]  L. Formaggia,et al.  Numerical modeling of 1D arterial networks coupled with a lumped parameters description of the heart , 2006, Computer methods in biomechanics and biomedical engineering.

[33]  E. Konofagou,et al.  A composite high-frame-rate system for clinical cardiovascular imaging , 2008, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[34]  W. Bautz,et al.  Clinical evaluation of bone-subtraction CT angiography (BSCTA) in head and neck imaging , 2006, European Radiology.

[35]  M. A. Casarin,et al.  Low-energy basis preconditioning for elliptic substructured solvers based on unstructured spectral/ hp element discretization , 2001 .

[36]  T F Sherman,et al.  On connecting large vessels to small. The meaning of Murray's law , 1981, The Journal of general physiology.

[37]  Michael Manhart,et al.  Vortex Shedding from a Hemisphere in a Turbulent Boundary Layer , 1998 .

[38]  Jan Mandel,et al.  Two-level domain decomposition preconditioning for the p-version finite element method in three dimensions , 1990 .

[39]  George Em Karniadakis,et al.  Distribution of WSS on the Internal Carotid Artery With an Aneurysm: A CFD Sensitivity Study , 2007 .

[40]  Chi-Wang Shu,et al.  Discontinuous Galerkin Methods: Theory, Computation and Applications , 2011 .

[41]  George Em Karniadakis,et al.  A triangular spectral element method; applications to the incompressible Navier-Stokes equations , 1995 .

[42]  Charles A. Taylor,et al.  Morphometry-Based Impedance Boundary Conditions for Patient-Specific Modeling of Blood Flow in Pulmonary Arteries , 2007, Annals of Biomedical Engineering.

[43]  S. Sherwin,et al.  One-dimensional modelling of a vascular network in space-time variables , 2003 .

[44]  Nadia Magnenat-Thalmann,et al.  The SPHERIGON: a simple polygon patch for smoothing quickly your polygonal meshes , 1998, Proceedings Computer Animation '98 (Cat. No.98EX169).

[45]  Robert W. Dutton,et al.  Level set methods and MR image segmentation for geometric modeling in computational hemodynamics , 1998, Proceedings of the 20th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Vol.20 Biomedical Engineering Towards the Year 2000 and Beyond (Cat. No.98CH36286).

[46]  S. Wandzurat,et al.  Symmetric quadrature rules on a triangle , 2003 .

[47]  Suchuan Dong,et al.  Dual-level parallelism for high-order CFD methods , 2004, Parallel Comput..

[48]  C. Putman,et al.  Patient-specific computational fluid dynamics modeling of anterior communicating artery aneurysms: a study of the sensitivity of intra-aneurysmal flow patterns to flow conditions in the carotid arteries. , 2006, AJNR. American journal of neuroradiology.

[49]  C. R. Ethier,et al.  Accuracy of Computational Hemodynamics in Complex Arterial Geometries Reconstructed from Magnetic Resonance Imaging , 2004, Annals of Biomedical Engineering.

[50]  Fotis Sotiropoulos,et al.  Curvilinear immersed boundary method for simulating fluid structure interaction with complex 3D rigid bodies , 2008, J. Comput. Phys..

[51]  D. Ku.  BLOOD FLOW IN ARTERIES , 1997 .

[52]  I. Bica,et al.  Iterative substructuring algorithms for the p-version finite element method for elliptic problems , 1997 .

[53]  S. Sherwin,et al.  Modelling the circle of Willis to assess the effects of anatomical variations and occlusions on cerebral flows. , 2007, Journal of biomechanics.

[54]  G. Karniadakis,et al.  Outflow Boundary Conditions for Arterial Networks with Multiple Outlets , 2008, Annals of Biomedical Engineering.

[55]  S. Orszag,et al.  High-order splitting methods for the incompressible Navier-Stokes equations , 1991 .

[56]  Charles A. Taylor,et al.  Outflow boundary conditions for three-dimensional finite element modeling of blood flow and pressure in arteries , 2006 .

[57]  Alex M. Andrew,et al.  Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science (2nd edition) , 2000 .

[58]  Céline Fouard,et al.  A Novel Three‐Dimensional Computer‐Assisted Method for a Quantitative Study of Microvascular Networks of the Human Cerebral Cortex , 2006, Microcirculation.

[59]  B. Weir.  Unruptured intracranial aneurysms: a review. , 2002, Journal of neurosurgery.

[60]  Marco Lúcio Bittencourt,et al.  Fully tensorial nodal and modal shape functions for triangles and tetrahedra , 2005 .

[61]  J P Shillingford,et al.  Pressure-flow relationships and vascular impedance in man. , 1970, Cardiovascular research.

[62]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[63]  Paul F. Fischer,et al.  Hybrid Multigrid/Schwarz Algorithms for the Spectral Element Method , 2005, J. Sci. Comput..

[64]  D. Holdsworth,et al.  Flow Patterns at the Stenosed Carotid Bifurcation: Effect of Concentric versus Eccentric Stenosis , 2000, Annals of Biomedical Engineering.

[65]  Shantenu Jha,et al.  NEKTAR, SPICE and Vortonics: using federated grids for large scale scientific applications , 2006, 2006 IEEE Challenges of Large Applications in Distributed Environments.

[66]  Alfio Quarteroni,et al.  Multiscale modelling of the circulatory system: a preliminary analysis , 1999 .

[67]  C Clark,et al.  The propagation of turbulence produced by a stenosis. , 1980, Journal of biomechanics.

[68]  P R Hoskins,et al.  MRI measurement of wall shear stress vectors in bifurcation models and comparison with CFD predictions , 2001, Journal of magnetic resonance imaging : JMRI.

[69]  Juan R Cebral,et al.  Patient-specific computational modeling of cerebral aneurysms with multiple avenues of flow from 3D rotational angiography images. , 2006, Academic radiology.

[70]  D. Ku,et al.  Pulsatile Flow and Atherosclerosis in the Human Carotid Bifurcation: Positive Correlation between Plaque Location and Low and Oscillating Shear Stress , 1985, Arteriosclerosis.

[71]  Mark G. Blyth,et al.  A Lobatto interpolation grid over the triangle , 2006 .

[72]  T. Hughes,et al.  Effect of exercise on hemodynamic conditions in the abdominal aorta. , 1999, Journal of vascular surgery.

[73]  P. Holmes,et al.  The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows , 1993 .

[74]  Dongbin Xiu,et al.  Parametric uncertainty analysis of pulse wave propagation in a model of a human arterial network , 2007, J. Comput. Phys..

[75]  P. Fischer,et al.  Simulation of high-Reynolds number vascular flows , 2007 .