Learning to map 2D ultrasound images into 3D space with minimal human annotation

In fetal neurosonography, aligning two-dimensional (2D) ultrasound scans to their corresponding plane in the three-dimensional (3D) space remains a challenging task. In this paper, we propose a convolutional neural network that predicts the position of 2D ultrasound fetal brain scans in 3D atlas space. Instead of purely supervised learning that requires heavy annotations for each 2D scan, we train the model by sampling 2D slices from 3D fetal brain volumes, and target the model to predict the inverse of the sampling process, resembling the idea of self-supervised learning. We propose a model that takes a set of images as input, and learns to compare them in pairs. The pairwise comparison is weighted by the attention module based on its contribution to the prediction, which is learnt implicitly during training. The feature representation for each image is thus computed by incorporating the relative position information to all the other images in the set, and is later used for the final prediction. We benchmark our model on 2D slices sampled from 3D fetal brain volumes at 18-22 weeks' gestational age. Using three evaluation metrics, namely, Euclidean distance, plane angles and normalized cross correlation, which account for both the geometric and appearance discrepancy between the ground-truth and prediction, in all these metrics, our model outperforms a baseline model by as much as 23%, when the number of input images increases. We further demonstrate that our model generalizes to (i) real 2D standard transthalamic plane images, achieving comparable performance as human annotations, as well as (ii) video sequences of 2D freehand fetal brain scans.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[3]  Hao Chen,et al.  Automatic Fetal Ultrasound Standard Plane Detection Using Knowledge Transferred Recurrent Neural Networks , 2015, MICCAI.

[4]  Jimmy Espinoza,et al.  Three‐ and 4‐Dimensional Ultrasound in Obstetric Practice , 2005, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[5]  Tae Hee Han,et al.  Fast Normalized Cross-Correlation , 2009, Circuits Syst. Signal Process..

[6]  Loïc Le Folgoc,et al.  Automatic View Planning with Multi-scale Deep Reinforcement Learning Agents , 2018, MICCAI.

[7]  T. Winter,et al.  The Cavum Septi Pellucidi , 2010, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[8]  D. Altman,et al.  International standards for fetal growth based on serial ultrasound measurements: the Fetal Growth Longitudinal Study of the INTERGROWTH-21st Project , 2014, The Lancet.

[9]  Liang Chen,et al.  Attention-Gated Networks for Improving Ultrasound Scan Plane Detection , 2018, MICCAI 2018.

[10]  Yuan Gao,et al.  Detection and Characterization of the Fetal Heartbeat in Free-hand Ultrasound Sweeps with Weakly-supervised Two-streams Convolutional Networks , 2017, MICCAI.

[11]  L. Chitty,et al.  Fetal Size and Dating: Charts Recommended for Clinical Obstetric Practice , 2009 .

[12]  Simon K. Warfield,et al.  Robust Super-Resolution Volume Reconstruction From Slice Acquisitions: Application to Fetal Brain MRI , 2010, IEEE Transactions on Medical Imaging.

[13]  Mary A. Rutherford,et al.  Reconstruction of fetal brain MRI with intensity matching and complete outlier removal , 2012, Medical Image Anal..

[14]  Hao Chen,et al.  Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks , 2015, IEEE Journal of Biomedical and Health Informatics.

[15]  G. Pilu,et al.  Transabdominal sonography of the cavum septum pellucidum in normal fetuses in the second and third trimesters of pregnancy , 2000, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[16]  B. Benacerraf,et al.  Three‐dimensional Fetal Sonography , 2002, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[17]  I. Timor-Tritsch,et al.  Sonographic examination of the fetal central nervous system: guidelines for performing the ‘basic examination’ and the ‘fetal neurosonogram’ , 2007, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[18]  Andrew Zisserman,et al.  Fully‐automated alignment of 3D fetal brain ultrasound to a canonical reference space using multi‐task learning , 2018, Medical Image Anal..

[19]  Tianfu Wang,et al.  Intelligent scanning: automated standard plane selection and biometric measurement of early gestational sac in routine ultrasound examination. , 2012, Medical physics.

[20]  Bishesh Khanal,et al.  Standard Plane Localisation in 3D Fetal Ultrasound Using Network with Geometric and Image Loss , 2018 .

[21]  Xin Yang,et al.  Standard plane localization in ultrasound by radial component , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[22]  Amir Alansary,et al.  Predicting Slice-to-Volume Transformation in Presence of Arbitrary Subject Motion , 2017, MICCAI.

[23]  Colin Studholme,et al.  Registration-based approach for reconstruction of high-resolution in utero fetal MR brain images. , 2006, Academic radiology.

[24]  Razvan Pascanu,et al.  A simple neural network module for relational reasoning , 2017, NIPS.

[25]  Daniel Rueckert,et al.  MRI of Moving Subjects Using Multislice Snapshot Images With Volume Reconstruction (SVR): Application to Fetal, Neonatal, and Adult Brain Studies , 2007, IEEE Transactions on Medical Imaging.

[26]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[27]  J. Alison Noble,et al.  Robust Regression of Brain Maturation from 3D Fetal Neurosonography Using CRNs , 2017, FIFI/OMIA@MICCAI.

[28]  Bartlomiej W. Papiez,et al.  Automated fetal brain extraction from clinical Ultrasound volumes using 3D Convolutional Neural Networks , 2019, MIUA.

[29]  Hao Qin,et al.  Agent with Warm Start and Active Termination for Plane Localization in 3D Ultrasound , 2019, MICCAI.

[30]  J. Alison Noble,et al.  Automated 3D Ultrasound Biometry Planes Extraction for First Trimester Fetal Assessment , 2016, MLMI@MICCAI.

[31]  Andrew Zisserman,et al.  Temporal HeartNet: Towards Human-Level Automatic Analysis of Fetal Cardiac Screening Video , 2017, MICCAI.

[32]  Xin Yang,et al.  Selective Search and Sequential Detection for Standard Plane Localization in Ultrasound , 2013, Abdominal Imaging.

[33]  L. D. de Vries,et al.  Grade and symmetry of normal fetal cortical development: a longitudinal two‐ and three‐dimensional ultrasound study , 2010, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[34]  Nikos Paragios,et al.  Slice-to-volume medical image registration: a survey , 2017, Medical Image Anal..

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Li Shen,et al.  Comparator Networks , 2018, ECCV.

[37]  D. Lev,et al.  Differential diagnosis in fetuses with absent septum pellucidum , 2005, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[38]  E. Alibrahim,et al.  A pictorial guide for the second trimester ultrasound , 2013, Australasian journal of ultrasound in medicine.