Improving PET-CT Image Segmentation via Deep Multi-modality Data Augmentation

Positron emission tomography (PET) - computed tomography (CT) is a widely-accepted imaging modality for staging, diagnosis and treatment response monitoring of cancers. Deep learning based computer aided diagnosis systems have achieved high accuracy on tumor segmentation on PET-CT images in recent years. PET images can be used to detect functional structures such as tumors, whilst CT images provide complementary anatomical information. As for tumor detection using deep learning methods, multi-modality segmentation was verified to be effective. In this work, we propose a generative adversarial network (GAN) based augmentation method to synthesized multi-modality data pairs on PET and CT to improve the training of multi-modality segmentation method. Our novelty lies in creating a semantic label augmentation method to provide latent information that is suitable for the multi-modality synthesis. In addition, we set out a ‘Split U’ structure which can generate both PET-CT modalities from a latent input. Our experimental results demonstrated that the synthesized images generated by our method can be used to augment the training data for PET-CT segmentation.