A Point Cloud Generative Model via Tree-Structured Graph Convolutions for 3D Brain Shape Reconstruction

Fusing medical images and the corresponding 3D shape representation can provide complementary information and microstructure details to improve the operational performance and accuracy in brain surgery. However, compared to the substantial image data, it is almost impossible to obtain the intraoperative 3D shape information by using physical methods such as sensor scanning, especially in minimally invasive surgery and robot-guided surgery. In this paper, a general generative adversarial network (GAN) architecture based on graph convolutional networks is proposed to reconstruct the 3D point clouds (PCs) of brains by using one single 2D image, thus relieving the limitation of acquiring 3D shape data during surgery. Specifically, a tree-structured generative mechanism is constructed to use the latent vector effectively and transfer features between hidden layers accurately. With the proposed generative model, a spontaneous image-to-PC conversion is finished in real-time. Competitive qualitative and quantitative experimental results have been achieved on our model. In multiple evaluation methods, the proposed model outperforms another common point cloud generative model PointOutNet.

[1]  Leonidas J. Guibas,et al.  Learning Representations and Generative Models for 3D Point Clouds , 2017, ICML.

[2]  Su-Lin Lee,et al.  Dynamic Shape Instantiation for Intra-operative Guidance , 2010, MICCAI.

[3]  Enrico Magli,et al.  Learning Localized Generative Models for 3D Point Clouds via Graph Convolution , 2018, ICLR.

[4]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[5]  K. Luk,et al.  Prediction of myelopathic level in cervical spondylotic myelopathy using diffusion tensor imaging , 2015, Journal of magnetic resonance imaging : JMRI.

[6]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[7]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[8]  A variational approach to nonlinear two-point boundary value problems , 2009 .

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Guang-Zhong Yang,et al.  One-stage Shape Instantiation from a Single 2D Image to 3D Point Cloud , 2019, MICCAI.

[11]  Junseok Kwon,et al.  3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Zhile Yang,et al.  Diabetic Retinopathy Diagnosis Using Multichannel Generative Adversarial Network With Semisupervision , 2020, IEEE Transactions on Automation Science and Engineering.

[13]  Xiaojuan Qi,et al.  GAL: Geometric Adversarial Loss for Single-View 3D-Object Reconstruction , 2018, ECCV.

[14]  Leonidas J. Guibas,et al.  FPNN: Field Probing Neural Networks for 3D Data , 2016, NIPS.

[15]  Shuqiang Wang,et al.  Medical Image Reconstruction Using Generative Adversarial Network for Alzheimer Disease Assessment with Class-Imbalance Problem , 2020, 2020 IEEE 6th International Conference on Computer and Communications (ICCC).

[16]  L. Rhines,et al.  A Novel Use of the Intraoperative MRI for Metastatic Spine Tumors: Laser Interstitial Thermal Therapy for Percutaneous Treatment of Epidural Metastatic Spine Disease. , 2017, Neurosurgery clinics of North America.

[17]  J. Blair,et al.  Open resection of hypothalamic hamartomas for intractable epilepsy revisited, using intraoperative MRI , 2018, Child's Nervous System.

[18]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jan S. Kirschke,et al.  Probabilistic Point Cloud Reconstructions for Vertebral Shape Analysis , 2019, MICCAI.

[20]  Shuqiang Wang,et al.  Brain MR to PET Synthesis via Bidirectional Generative Adversarial Network , 2020, MICCAI.

[21]  Wei Liu,et al.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images , 2018, ECCV.

[22]  Shu-Qiang Wang,et al.  A variational approach to nonlinear two-point boundary value problems , 2009, Comput. Math. Appl..

[23]  Ji-Huan He,et al.  Variational Iteration Method for a Nonlinear Reaction-Diffusion Process , 2008 .

[24]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[25]  Michael K. Ng,et al.  Tensorizing GAN With High-Order Pooling for Alzheimer’s Disease Assessment , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[27]  Jing Qin,et al.  Skin lesion segmentation via generative adversarial networks with dual discriminators , 2020, Medical Image Anal..

[28]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[30]  Dong Tian,et al.  FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Yin Zhou,et al.  VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[33]  Yong Hu,et al.  Skeletal Maturity Recognition Using a Fully Automated System With Convolutional Neural Networks , 2018, IEEE Access.

[34]  Yong Hu,et al.  Bone age assessment using convolutional neural networks , 2018, 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD).

[35]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[36]  Yong Hu,et al.  Classification of Diffusion Tensor Metrics for the Diagnosis of a Myelopathic Cord Using Machine Learning , 2017, Int. J. Neural Syst..

[37]  王书强.  基于3D卷积神经网络的区域降雨量预报3D Convolutional Neural Network for Regional Precipitation Nowcasting , 2018 .

[38]  Xiangyu Wang,et al.  Automatic Recognition of Mild Cognitive Impairment and Alzheimers Disease Using Ensemble based 3D Densely Connected Convolutional Networks , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[39]  Shuqiang Wang,et al.  A GA-based feature selection and parameter optimization for support tucker machine , 2017 .

[40]  Jan Dirk Wegner,et al.  Contour Detection in Unstructured 3D Point Clouds , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Harshad Rai,et al.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks , 2018 .