Tunable CT Lung Nodule Synthesis Conditioned on Background Image and Semantic Features

Synthetic CT image with artificially generated lung nodules has been shown to be useful as an augmentation method for certain tasks such as lung segmentation and nodule classification. Most conventional methods are designed as “inpainting” tasks by removing a region from background image and synthesizing the foreground nodule. To ensure natural blending with the background, existing method proposed loss function and separate shape/appearance generation. However, spatial discontinuity is still unavoidable for certain cases. Meanwhile, there is often little control over semantic features regarding the nodule characteristics, which may limit their capability of fine-grained augmentation in balancing the original data. In this work, we address these two challenges by developing a 3D multi-conditional generative adversarial network (GAN) that is conditioned on both background image and semantic features for lung nodule synthesis on CT image. Instead of removing part of the input image, we use a fusion block to blend object and background, ensuring more realistic appearance. Multiple discriminator scenarios are considered, and three outputs of image, segmentation, and feature are used to guide the synthesis process towards semantic feature control. We trained our method on public dataset, and showed promising results as a solution for tunable lung nodule synthesis.

[1]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[2]  Ronald M. Summers,et al.  Progressive and Multi-path Holistically Nested Neural Networks for Pathological Lung Segmentation from CT Images , 2017, MICCAI.

[3]  Nojun Kwak,et al.  MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis , 2018, BMVC.

[4]  Jeffrey L. Gunter,et al.  Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks , 2018, SASHIMI@MICCAI.

[5]  Youbao Tang,et al.  CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation , 2018, MICCAI.

[6]  Jie Yang,et al.  Decompose to manipulate: Manipulable Object Synthesis in 3D Medical Images with Structured Image Decomposition , 2018, ArXiv.

[7]  N. Müller,et al.  Fleischner Society: glossary of terms for thoracic imaging. , 2008, Radiology.

[8]  Jie Yang,et al.  Class-Aware Adversarial Lung Nodule Synthesis In CT Images , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[9]  Adam P. Harrison,et al.  3D Convolutional Neural Networks with Graph Refinement for Airway Segmentation Using Incomplete Data Labels , 2017, MLMI@MICCAI.

[10]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Ronald M. Summers,et al.  ChestX-ray: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly Supervised Classification and Localization of Common Thorax Diseases , 2019, Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics.

[12]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[13]  Richard C. Pais,et al.  The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. , 2011, Medical physics.