Real-Time Detection and Localisation of Fetal Standard Scan Planes in 2D Freehand Ultrasound

Identifying and interpreting fetal standard scan planes during 2D ultrasound mid-pregnancy examinations are highly complex tasks which require years of training. Apart from guiding the probe to the correct location, it can be equally difficult for a non-expert to identify relevant structures within the image. Automatic image processing can provide tools to help experienced as well as inexperienced operators with these tasks. In this paper, we propose a novel method based on convolutional neural networks which can automatically detect 13 fetal standard views in freehand 2D ultrasound data as well as provide a localisation of the fetal structures via a bounding box. An important contribution is that the network learns to localise the target anatomy using weak supervision only. The network architecture is designed to operate in real-time while providing optimal output for the localisation task. We present results for real-time annotation, retrospective frame retrieval from saved videos, and localisation on a very large and challenging dataset consisting of images and video recordings of full clinical anomaly screenings. The proposed method annotated video frames with an average F1-score of 0.86, and obtained a 90.09% accuracy for retrospective frame retrieval. Moreover, we achieved an accuracy of 77.8% on the localisation task.

[1]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[2]  Ivan Laptev,et al.  Is object localization for free? - Weakly-supervised learning with convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Hao Chen,et al.  Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks , 2015, IEEE Journal of Biomedical and Health Informatics.

[4]  Konstantinos Kamnitsas,et al.  Real-Time Standard Scan Plane Detection and Localisation in Fetal Ultrasound Using Fully Convolutional Neural Networks , 2016, MICCAI.

[5]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[6]  Ronan Collobert,et al.  From image-level to pixel-level labeling with Convolutional Networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[8]  Tianfu Wang,et al.  Intelligent scanning: automated standard plane selection and biometric measurement of early gestational sac in routine ultrasound examination. , 2012, Medical physics.

[9]  A. El-Zaart.  Images thresholding using ISODATA technique with gamma distribution , 2010, Pattern Recognition and Image Analysis.

[10]  Hao Chen,et al.  Automatic Fetal Ultrasound Standard Plane Detection Using Knowledge Transferred Recurrent Neural Networks , 2015, MICCAI.

[11]  Kaiqi Huang,et al.  Weakly Supervised Large Scale Object Localization with Multiple Instance Learning and Bag Splitting , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Z Alfirevic,et al.  Practice guidelines for performance of the routine mid‐trimester fetal ultrasound scan , 2011, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[13]  D. Sahota,et al.  Volumetric (3D) imaging reduces inter‐ and intraobserver variation of fetal biometry measurements , 2009, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[14]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Xin Yang,et al.  Selective Search and Sequential Detection for Standard Plane Localization in Ultrasound , 2013, Abdominal Imaging.

[17]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[18]  A. Abuhamad,et al.  Automated retrieval of standard diagnostic fetal cardiac ultrasound planes in the second trimester of pregnancy: a prospective evaluation of software , 2007, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[19]  J. Alison Noble,et al.  Automated annotation and quantitative description of ultrasound videos of the fetal heart , 2017, Medical Image Anal..

[20]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[21]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[22]  J. Alison Noble,et al.  Object localisation in fetal ultrasound images using invariant features , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[23]  Seetha Hari,et al.  Learning From Imbalanced Data , 2019, Advances in Computer and Electrical Engineering.

[24]  Zhi-Hua Zhou,et al.  Ieee Transactions on Knowledge and Data Engineering 1 Training Cost-sensitive Neural Networks with Methods Addressing the Class Imbalance Problem , 2022 .

[25]  J. Alison Noble,et al.  Guided Random Forests for Identification of Key Fetal Anatomy and Image Categorization in Ultrasound Scans , 2015, MICCAI.

[26]  J. Alison Noble,et al.  Fisher vector encoding for detecting objects of interest in ultrasound videos , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[27]  G. Hill,et al.  Disparities in the prenatal detection of critical congenital heart disease , 2015, Prenatal diagnosis.

[28]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[29]  J. Alison Noble,et al.  Searching for Structures of Interest in an Ultrasound Video Sequence , 2014, MLMI.

[30]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[31]  Xin Yang,et al.  Standard plane localization in ultrasound by radial component model and selective search. , 2014, Ultrasound in medicine & biology.