Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detection

In this work we present a novel system for generation of virtual PET images using CT scans. We combine a fully convolutional network (FCN) with a conditional generative adversarial network (GAN) to generate simulated PET data from given input CT data. The synthesized PET can be used for false-positive reduction in lesion detection solutions. Clinically, such solutions may enable lesion detection and drug treatment evaluation in a CT-only environment, thus reducing the need for the more expensive and radioactive PET/CT scan. Our dataset includes 60 PET/CT scans from Sheba Medical center. We used 23 scans for training and 37 for testing. Different schemes to achieve the synthesized output were qualitatively compared. Quantitative evaluation was conducted using an existing lesion detection software, combining the synthesized PET as a false positive reduction layer for the detection of malignant lesions in the liver. Current results look promising showing a 28% reduction in the average false positive per case from 2.9 to 2.1. The suggested solution is comprehensive and can be expanded to additional body organs, and different modalities.

[1]  Ieee Xplore,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Information for Authors , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Hayit Greenspan,et al.  Fully Convolutional Network for Liver Segmentation and Lesions Detection , 2016, LABELS/DLMIA@MICCAI.

[3]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[4]  Charles E Metz,et al.  Receiver operating characteristic analysis: a tool for the quantitative evaluation of observer performance and imaging systems. , 2006, Journal of the American College of Radiology : JACR.

[5]  Sebastian J. Schlecht,et al.  Automatic Liver and Tumor Segmentation of CT and MRI Volumes using Cascaded Fully Convolutional Neural Networks , 2017, ArXiv.

[6]  Hayit Greenspan,et al.  Virtual PET Images from CT Data Using Deep Convolutional Networks: Initial Results , 2017, SASHIMI@MICCAI.

[7]  Sotirios A. Tsaftaris,et al.  Adversarial Image Synthesis for Unpaired Multi-modal Cardiac Data , 2017, SASHIMI@MICCAI.

[8]  Edward H. Adelson,et al.  PYRAMID METHODS IN IMAGE PROCESSING. , 1984 .

[9]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[10]  Jelmer M. Wolterink,et al.  Deep MR to CT Synthesis Using Unpaired Data , 2017, SASHIMI@MICCAI.

[11]  Qian Wang,et al.  Deep embedding convolutional neural network for synthesizing CT image from T1‐Weighted MR image , 2018, Medical Image Anal..

[12]  R L Wahl,et al.  Does FDG uptake measure proliferative activity of human cancer cells? In vitro comparison with DNA flow cytometry and tritiated thymidine uptake. , 1993, Journal of nuclear medicine : official publication, Society of Nuclear Medicine.

[13]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  Hayit Greenspan,et al.  Fully convolutional network and sparsity-based dictionary learning for liver lesion detection in CT examinations , 2018, Neurocomputing.

[16]  Wolfgang A Weber,et al.  Technology Insight: advances in molecular imaging and an appraisal of PET/CT scanning , 2008, Nature Clinical Practice Oncology.

[17]  Paul Kinahan,et al.  Positron emission tomography-computed tomography standardized uptake values in clinical practice and assessing response to therapy. , 2010, Seminars in ultrasound, CT, and MR.

[18]  David Dagan Feng,et al.  Synthesis of Positron Emission Tomography (PET) Images via Multi-channel Generative Adversarial Networks (GANs) , 2017, CMMI/RAMBO/SWITCH@MICCAI.

[19]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[21]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[22]  Lale Kostakoglu,et al.  Clinical role of FDG PET in evaluation of cancer patients. , 2003, Radiographics : a review publication of the Radiological Society of North America, Inc.

[23]  Wolfgang A Weber,et al.  Assessing Tumor Response to Therapy , 2009, Journal of Nuclear Medicine.

[24]  Bruce D Cheson,et al.  Progress and Promise of FDG-PET Imaging for Cancer Patient Management and Oncologic Drug Development , 2005, Clinical Cancer Research.

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  Xiao Han,et al.  MR‐based synthetic CT generation using a deep convolutional neural network method , 2017, Medical physics.

[27]  Yaozong Gao,et al.  Estimating CT Image from MRI Data Using 3D Fully Convolutional Networks , 2016, LABELS/DLMIA@MICCAI.