X-ray2Shape: Reconstruction of 3D Liver Shape from a Single 2D Projection Image

Computed tomography (CT) and magnetic resonance imaging (MRI) scanners measure three-dimensional (3D) images of patients. However, only low-dimensional local two-dimensional (2D) images may be obtained during surgery or radiotherapy. Although computer vision techniques have shown that 3D shapes can be estimated from multiple 2D images, shape reconstruction from a single 2D image such as an endoscopic image or an X-ray image remains a challenge. In this study, we propose X-ray2Shape, which permits a deep learning-based 3D organ mesh to be reconstructed from a single 2D projection image. The method learns the mesh deformation from a mean template and deep features computed from the individual projection images. Experiments with organ meshes and digitally reconstructed radiograph (DRR) images of abdominal regions were performed to confirm the estimation performance of the methods.

[1]  Tetsuya Matsuda,et al.  Statistical Deformation Reconstruction Using Multi-organ Shape Features for Pancreatic Cancer Localization , 2019, Medical Image Anal..

[2]  Jinah Park,et al.  Hippocampal Shape Modeling Based on a Progressive Template Surface Deformation and its Verification , 2015, IEEE Transactions on Medical Imaging.

[3]  Wei Liu,et al.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images , 2018, ECCV.

[4]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[5]  Akira Saito,et al.  [POSTER] Deformation Estimation of Elastic Bodies Using Multiple Silhouette Images for Endoscopic Image Augmentation , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[6]  Nelson Lam,et al.  Radiation dose from cone beam computed tomography for image-guided radiation therapy. , 2008, International Journal of Radiation Oncology, Biology, Physics.

[7]  David A Jaffray,et al.  Patient dose from kilovoltage cone beam computed tomography imaging in radiation therapy. , 2006, Medical physics.

[8]  Yifan Wang,et al.  DeepOrganNet: On-the-Fly Reconstruction and Visualization of 3D / 4D Lung Models from Single-View Projections by Deep Deformation Network , 2019, IEEE Transactions on Visualization and Computer Graphics.

[9]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[10]  Tetsuya Matsuda,et al.  Reconstructing 3D Lung Shape from a Single 2D Image during the Deaeration Deformation Process using Model-based Data Augmentation , 2019, 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[11]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.