Deep learning based 3-dimensional liver motion estimation using 2-dimensional ultrasound images

Purpose: Radiofrequency ablation therapy support system aims to display the true tumor position by tracking the tumor during the treatment. This sys-tem utilizes 2-dimensional ultrasound images. As a major cause of the tracking error, the tumor moves in the direction which is perpendicular to the scan plane of the ultra-sound image. Methods: To cope with this, in the proposed method, the six-axis movement amount is estimated from two 2-dimensional ultrasound images by a convolution neural network. In addition, we tried to improve the processing speed by using multitask learning. The liver moves differently from other tissues in the ultra-sound image. In particular, in order to improve the accuracy, a heatmap as a result of segmentation estimation of the liver region was utilized for estimating the amount of movement. By this method, we could estimate the amount of movement in the direction which is perpendicular to the ultrasound image plane so as to track the organ motion more precisely and robustly. We conducted an experiment to confirm the estimating performance of the amount of probe movement relative to the liver phantom, and evaluated estimation accuracy and measured processing speed. Results: The number of the used image sets was 9,506 for the learning and 1,337 for the evaluation. Conclusion: The experimental results show the effectiveness of our novel method. This article does not contain any studies with human participants or animals performed by any of the authors.

[1]  Mehrdad Salehi,et al.  Deep Learning for Sensorless 3D Freehand Ultrasound Imaging , 2017, MICCAI.

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Emil Slusanschi,et al.  A Multi-Stage Multi-Task Neural Network for Aerial Scene Interpretation and Geolocalization , 2018, ArXiv.

[4]  Takafumi Aoki,et al.  Probe localization using structure from motion for 3D ultrasound image reconstruction , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[5]  Roberto Cipolla,et al.  PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Jean-Arcady Meyer,et al.  Fast and Incremental Method for Loop-Closure Detection Using Bags of Visual Words , 2008, IEEE Transactions on Robotics.

[7]  Hao Qin,et al.  Loop closure detection in SLAM by combining visual CNN features and submaps , 2018, 2018 4th International Conference on Control, Automation and Robotics (ICCAR).

[8]  J. M. M. Montiel,et al.  Visual SLAM for Handheld Monocular Endoscope , 2014, IEEE Transactions on Medical Imaging.

[9]  Annette Stahl,et al.  Convolutional Autoencoder aided loop closure detection for monocular SLAM , 2018 .

[10]  Cristian Sminchisescu,et al.  Deep Multitask Architecture for Integrated 2D and 3D Human Sensing , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Jocelyne Troccaz,et al.  3D-2D ultrasound feature-based registration for navigated prostate biopsy: A feasibility study , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[12]  Yan Su,et al.  Loop closure detection for visual SLAM systems using convolutional neural network , 2017, 2017 23rd International Conference on Automation and Computing (ICAC).

[13]  Norihiro Koizumi,et al.  An automatic templates selection method for ultrasound guided tumor tracking , 2017, 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[14]  Yasin Almalioglu,et al.  A Deep Learning Based 6 Degree-of-Freedom Localization Method for Endoscopic Capsule Robots , 2017, ArXiv.

[15]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[16]  Georgios Tzimiropoulos,et al.  Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Kiyoharu Aizawa,et al.  Joint Optimization Framework for Learning with Noisy Labels , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).