Using GPUs to accelerate computational diffusion MRI: From microstructure estimation to tractography and connectomes

ABSTRACT The great potential of computational diffusion MRI (dMRI) relies on indirect inference of tissue microstructure and brain connections, since modelling and tractography frameworks map diffusion measurements to neuroanatomical features. This mapping however can be computationally highly expensive, particularly given the trend of increasing dataset sizes and the complexity in biophysical modelling. Limitations on computing resources can restrict data exploration and methodology development. A step forward is to take advantage of the computational power offered by recent parallel computing architectures, especially Graphics Processing Units (GPUs). GPUs are massive parallel processors that offer trillions of floating point operations per second, and have made possible the solution of computationally‐intensive scientific problems that were intractable before. However, they are not inherently suited for all problems. Here, we present two different frameworks for accelerating dMRI computations using GPUs that cover the most typical dMRI applications: a framework for performing biophysical modelling and microstructure estimation, and a second framework for performing tractography and long‐range connectivity estimation. The former provides a front‐end and automatically generates a GPU executable file from a user‐specified biophysical model, allowing accelerated non‐linear model fitting in both deterministic and stochastic ways (Bayesian inference). The latter performs probabilistic tractography, can generate whole‐brain connectomes and supports new functionality for imposing anatomical constraints, such as inherent consideration of surface meshes (GIFTI files) along with volumetric images. We validate the frameworks against well‐established CPU‐based implementations and we show that despite the very different challenges for parallelising these problems, a single GPU achieves better performance than 200 CPU cores thanks to our parallel designs. HIGHLIGHTSThe computational power offered by GPUs is used to accelerate the analysis of dMRI.We present a generic and flexible parallel framework for microstructure modelling on GPUs.And also a framework for performing probabilistic tractography and generating connectomes on GPUs.A single GPU achieves better performance than 200 CPU cores with our frameworks.

[1]  P. Matthews,et al.  Multimodal population brain imaging in the UK Biobank prospective epidemiological study , 2016, Nature Neuroscience.

[2]  H J Motulsky,et al.  Fitting curves to data using nonlinear regression: a practical and nonmathematical review , 1987, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[3]  Jean-Philippe Thiran,et al.  Accelerated Microstructure Imaging via Convex Optimization (AMICO) from diffusion MRI data , 2015, NeuroImage.

[4]  Chad J. Donahue,et al.  Using Diffusion Tractography to Predict Cortical Connection Strength and Distance: A Quantitative Comparison with Tracers in the Monkey , 2016, The Journal of Neuroscience.

[5]  Yu Wang,et al.  Probabilistic Brain Fiber Tractography on GPUs , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum.

[6]  Steen Moeller,et al.  Multiband multislice GE‐EPI at 7 tesla, with 16‐fold acceleration using partial parallel imaging with application to high spatial and temporal whole‐brain fMRI , 2010, Magnetic resonance in medicine.

[7]  Timothy Edward John Behrens,et al.  Changes in connectivity profiles define functionally distinct regions in human medial frontal cortex. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[8]  Eros Comunello,et al.  Diffusion tensor fiber tracking on graphics processing units , 2008, Comput. Medical Imaging Graph..

[9]  Rachid Deriche,et al.  Mipy: An Open-Source Framework to improve reproducibility in Brain Microstructure Imaging , 2018 .

[10]  David J. C. MacKay,et al.  Developments in Probabilistic Modelling with Neural Networks - Ensemble Learning , 1995, SNN Symposium on Neural Networks.

[11]  V. Wedeen,et al.  Fiber crossing in human brain depicted with diffusion tensor MR imaging. , 2000, Radiology.

[12]  Jennifer A McNab,et al.  Sensitivity of diffusion weighted steady state free precession to anisotropic diffusion , 2008, Magnetic resonance in medicine.

[13]  Daniel C. Alexander,et al.  Crossing Versus Fanning: Model Comparison Using HCP Data , 2016 .

[14]  Lin-Ching Chang,et al.  GPU acceleration of nonlinear diffusion tensor estimation using CUDA and MPI , 2014, Neurocomputing.

[15]  Hui Zhang,et al.  Imaging brain microstructure with diffusion MRI: practicality and applications , 2019, NMR in biomedicine.

[16]  Steen Moeller,et al.  Advances in diffusion MRI acquisition and processing in the Human Connectome Project , 2013, NeuroImage.

[17]  Kevin Skadron,et al.  Scalable parallel programming , 2008, 2008 IEEE Hot Chips 20 Symposium (HCS).

[18]  Timothy Edward John Behrens,et al.  Characterization and propagation of uncertainty in diffusion‐weighted MR imaging , 2003, Magnetic resonance in medicine.

[19]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[20]  D. Norris,et al.  Biexponential diffusion attenuation in various states of brain tissue: Implications for diffusion‐weighted imaging , 1996, Magnetic resonance in medicine.

[21]  Stamatios N. Sotiropoulos,et al.  Improved fibre dispersion estimation using b-tensor encoding , 2019, NeuroImage.

[22]  採編典藏組 Society for Industrial and Applied Mathematics(SIAM) , 2008 .

[23]  Rainer Goebel,et al.  Robust and fast nonlinear optimization of diffusion MRI microstructure models , 2017, NeuroImage.

[24]  Christopher Rorden,et al.  Image Processing and Quality Control for the first 10,000 Brain Imaging Datasets from UK Biobank , 2017 .

[25]  J. Polimeni,et al.  Blipped‐controlled aliasing in parallel imaging for simultaneous multislice echo planar imaging with reduced g‐factor penalty , 2012, Magnetic resonance in medicine.

[26]  Michael J. Flynn,et al.  Some Computer Organizations and Their Effectiveness , 1972, IEEE Transactions on Computers.

[27]  Stamatios N. Sotiropoulos,et al.  XTRACT - Standardised protocols for automated tractography in the human and macaque brain , 2020, NeuroImage.

[28]  Y. Cohen,et al.  Non-mono-exponential attenuation of water and N-acetyl aspartate signals due to diffusion in brain tissue. , 1998, Journal of magnetic resonance.

[29]  Anders Eklund,et al.  Medical image processing on the GPU - Past, present and future , 2013, Medical Image Anal..

[30]  Mark W. Woolrich,et al.  Advances in functional and structural MR image analysis and implementation as FSL , 2004, NeuroImage.

[31]  Manuel Graña,et al.  Model‐based analysis of multishell diffusion MR data for tractography: How to get over fitting problems , 2012, Magnetic resonance in medicine.

[32]  Carl Tim Kelley,et al.  Iterative methods for optimization , 1999, Frontiers in applied mathematics.

[33]  Alan Edelman,et al.  The efficient evaluation of the hypergeometric function of a matrix argument , 2006, Math. Comput..

[34]  David C. Van Essen,et al.  The future of the human connectome , 2012, NeuroImage.

[35]  A. Wood,et al.  Saddlepoint approximations for the Bingham and Fisher–Bingham normalising constants , 2005 .

[36]  D. Parker,et al.  Analysis of partial volume effects in diffusion‐tensor MRI , 2001, Magnetic resonance in medicine.

[37]  P. Basser,et al.  Water Diffusion Changes in Wallerian Degeneration and Their Dependence on White Matter Architecture , 2000 .

[38]  Alex Fit-Florea,et al.  Precision and Performance: Floating Point and IEEE 754 Compliance for NVIDIA GPUs , 2011 .

[39]  Andrew Zalesky,et al.  Building connectomes using diffusion MRI: why, how and but , 2017, NMR in biomedicine.

[40]  Frank Lindseth,et al.  Medical image segmentation on GPUs - A comprehensive review , 2015, Medical Image Anal..

[41]  P. Basser,et al.  Axcaliber: A method for measuring axon diameter distribution from diffusion MRI , 2008, Magnetic resonance in medicine.

[42]  Michael A. Saunders,et al.  Procedures for optimization problems with a mixture of bounds and general linear constraints , 1984, ACM Trans. Math. Softw..

[43]  Daniel C. Alexander,et al.  Multiple Fibers: Beyond the Diffusion Tensor , 2013 .

[44]  Steen Moeller,et al.  The Human Connectome Project: A data acquisition perspective , 2012, NeuroImage.

[45]  Gabor T. Marth,et al.  A global reference for human genetic variation , 2015, Nature.

[46]  J C Gore,et al.  Diffusion‐weighted imaging in tissues: Theoretical models , 1995, NMR in biomedicine.

[47]  Xiaoping Hu,et al.  The effects of connection reconstruction method on the interregional connectivity of brain networks via diffusion tractography , 2012, Human brain mapping.

[48]  Alard Roebroeck,et al.  Robust and fast Monte Carlo Markov Chain sampling of diffusion MRI microstructure models , 2018, bioRxiv.

[49]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[50]  Ben Jeurissen,et al.  Diffusion MRI fiber tractography of the brain , 2019, NMR in biomedicine.

[51]  Mark E. Bastin,et al.  Peak Width of Skeletonized Water Diffusion MRI in the Neonatal Brain , 2020, Frontiers in Neurology.

[52]  Barbara Chapman,et al.  Using OpenMP - portable shared memory parallel programming , 2007, Scientific and engineering computation.

[53]  José M. García,et al.  Accelerating Fibre Orientation Estimation from Diffusion Weighted Magnetic Resonance Imaging Using GPUs , 2012, PDP.

[54]  Stephen M Smith,et al.  Fast robust automated brain extraction , 2002, Human brain mapping.

[55]  Daniel C. Alexander,et al.  NODDI: Practical in vivo neurite orientation dispersion and density imaging of the human brain , 2012, NeuroImage.

[56]  Stefan Klein,et al.  Fast parallel image registration on CPU and GPU for diagnostic classification of Alzheimer's disease , 2013, Front. Neuroinform..

[57]  Sean C L Deoni,et al.  Quantitative Relaxometry of the Brain , 2010, Topics in magnetic resonance imaging : TMRI.

[58]  John E. Stone,et al.  OpenCL: A Parallel Programming Standard for Heterogeneous Computing Systems , 2010, Computing in Science & Engineering.

[59]  Daniel C. Alexander,et al.  Bingham–NODDI: Mapping anisotropic orientation dispersion of neurites using diffusion MRI , 2016, NeuroImage.

[60]  A. Szafer,et al.  An analytical model of restricted diffusion in bovine optic nerve , 1997, Magnetic resonance in medicine.

[61]  Mark E. Bastin,et al.  Neonatal morphometric similarity mapping for predicting brain age and characterizing neuroanatomic variation associated with preterm birth , 2020, NeuroImage: Clinical.

[62]  Justin P. Haldar,et al.  Accelerating advanced mri reconstructions on gpus , 2008, CF '08.

[63]  Thomas J. Grabowski,et al.  Running Neuroimaging Applications on Amazon Web Services: How, When, and at What Cost? , 2017, Front. Neuroinform..

[64]  Olaf Sporns,et al.  The Human Connectome: A Structural Description of the Human Brain , 2005, PLoS Comput. Biol..

[65]  William H. Press,et al.  Numerical Recipes in FORTRAN - The Art of Scientific Computing, 2nd Edition , 1987 .

[66]  P. Batchelor,et al.  International Society for Magnetic Resonance in Medicine , 1997 .

[67]  P. Basser,et al.  MR diffusion tensor spectroscopy and imaging. , 1994, Biophysical journal.

[68]  Joseph O'Rourke,et al.  Computational Geometry in C: Search and Intersection , 1998 .

[69]  Stefan Klein,et al.  Improving alignment in Tract-based spatial statistics: Evaluation and optimization of image registration , 2013, NeuroImage.

[70]  M. Jenkinson Non-linear registration aka Spatial normalisation , 2007 .

[71]  Gabor T. Marth,et al.  An integrated map of structural variation in 2,504 human genomes , 2015, Nature.

[72]  Tim B. Dyrby,et al.  Orientationally invariant indices of axon diameter and density from diffusion MRI , 2010, NeuroImage.

[73]  Hui Zhang,et al.  Axon diameter mapping in the presence of orientation dispersion with diffusion MRI , 2011, NeuroImage.

[74]  Timothy Edward John Behrens,et al.  Ball and rackets: Inferring fiber fanning from diffusion-weighted MRI , 2012, NeuroImage.

[75]  P. Basser,et al.  Estimation of the effective self-diffusion tensor from the NMR spin echo. , 1994, Journal of magnetic resonance. Series B.

[76]  S.N. Sotiropoulos,et al.  High resolution whole brain diffusion imaging at 7T for the Human Connectome Project , 2015, NeuroImage.

[77]  Timothy Edward John Behrens,et al.  Accelerating Fibre Orientation Estimation from Diffusion Weighted Magnetic Resonance Imaging Using GPUs , 2012, 2012 20th Euromicro International Conference on Parallel, Distributed and Network-based Processing.

[78]  Anders Eklund,et al.  BROCCOLI: Software for fast fMRI analysis on many-core CPUs and GPUs , 2014, Front. Neuroinform..

[79]  Lawrence L. Wald,et al.  White matter compartment models for in vivo diffusion MRI at 300mT/m , 2015, NeuroImage.

[80]  P. Basser Diffusion MRI: From Quantitative Measurement to In vivo Neuroanatomy , 2009 .

[81]  Stephen M. Smith,et al.  Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm , 2001, IEEE Transactions on Medical Imaging.

[82]  Andac Hamamci,et al.  Cellular Automata Tractography: Fast Geodesic Diffusion MR Tractography and Connectivity Based Segmentation on the GPU , 2019, Neuroinformatics.

[83]  Mark Jenkinson,et al.  The minimal preprocessing pipelines for the Human Connectome Project , 2013, NeuroImage.

[84]  C. Westin,et al.  Multi‐component apparent diffusion coefficients in human brain † , 1999, NMR in biomedicine.

[85]  P. Basser,et al.  In vivo fiber tractography using DT‐MRI data , 2000, Magnetic resonance in medicine.

[86]  C. Poupon,et al.  Regularization of Diffusion-Based Direction Maps for the Tracking of Brain White Matter Fascicles , 2000, NeuroImage.

[87]  Mahmoud Al-Ayyoub,et al.  Accelerating compute intensive medical imaging segmentation algorithms using hybrid CPU-GPU implementations , 2017, Multimedia Tools and Applications.

[88]  Alan Connelly,et al.  Anatomically-constrained tractography: Improved diffusion MRI streamlines tractography through effective use of anatomical information , 2012, NeuroImage.

[89]  Mark W. Woolrich,et al.  Probabilistic diffusion tractography with multiple fibre orientations: What can we gain? , 2007, NeuroImage.

[90]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[91]  Yogesh Rathi,et al.  High‐resolution in vivo diffusion imaging of the human brain with generalized slice dithered enhanced resolution: Simultaneous multislice (gSlider‐SMS) , 2018, Magnetic resonance in medicine.

[92]  Graham Pullan,et al.  BarraCUDA - a fast short read sequence aligner using graphics processing units , 2011, BMC Research Notes.