Mask-conditioned latent diffusion for generating gastrointestinal polyp images

In order to take advantage of artificial intelligence (AI) solutions in endoscopy diagnostics, we must overcome the issue of limited annotations. These limitations are caused by the high privacy concerns in the medical field and the requirement of getting aid from experts for the time-consuming and costly medical data annotation process. In computer vision, image synthesis has made a significant contribution in recent years, as a result of the progress of generative adversarial networks (GANs) and diffusion probabilistic models (DPMs). Novel DPMs have outperformed GANs in text, image, and video generation tasks. Therefore, this study proposes a conditional DPM framework to generate synthetic gastrointestinal (GI) polyp images conditioned on given generated segmentation masks. Our experimental results show that our system can generate an unlimited number of high-fidelity synthetic polyp images with the corresponding ground truth masks of polyps. To test the usefulness of the generated data we trained binary image segmentation models to study the effect of using synthetic data. Results show that the best micro-imagewise intersection over union (IOU) of 0.7751 was achieved from DeepLabv3+ when the training data consists of both real data and synthetic data. However, the results reflect that achieving good segmentation performance with synthetic data heavily depends on model architectures.

[1]  Han Zhang,et al.  Examining the effect of synthetic data augmentation in polyp detection and segmentation , 2022, International Journal of Computer Assisted Radiology and Surgery.

[2]  Vajira Lasantha Thambawita,et al.  PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps , 2022, 2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS).

[3]  L. Szpruch,et al.  Synthetic Data - what, why and how? , 2022, ArXiv.

[4]  I. Balasingham,et al.  Simple U-net based synthetic polyp image generation: Polyp to negative and negative to polyp , 2022, Biomed. Signal Process. Control..

[5]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Jørgen K. Kanters,et al.  DeepFake electrocardiograms using generative adversarial networks are the beginning of the end for privacy issues in medicine , 2021, Scientific Reports.

[7]  Jenia Jitsev,et al.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs , 2021, ArXiv.

[8]  Sajad Amouei Sheshkal,et al.  SinGAN-Seg: Synthetic training data generation for medical image segmentation , 2021, PloS one.

[9]  Vajira Lasantha Thambawita,et al.  ID: 3523524 DATA AUGMENTATION USING GENERATIVE ADVERSARIAL NETWORKS FOR CREATING REALISTIC ARTIFICIAL COLON POLYP IMAGES: VALIDATION STUDY BY ENDOSCOPISTS , 2021 .

[10]  Ming Y. Lu,et al.  Synthetic data in machine learning for medicine and healthcare , 2021, Nature Biomedical Engineering.

[11]  Jørgen K. Kanters,et al.  DeepSynthBody: the beginning of the end for data deficiency in medicine , 2021, 2021 International Conference on Applied Artificial Intelligence (ICAPAI).

[12]  Prafulla Dhariwal,et al.  Diffusion Models Beat GANs on Image Synthesis , 2021, NeurIPS.

[13]  Prafulla Dhariwal,et al.  Improved Denoising Diffusion Probabilistic Models , 2021, ICML.

[14]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[15]  Vajira Lasantha Thambawita,et al.  An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning Applied to Gastrointestinal Tract Abnormality Classification , 2020, ACM Trans. Comput. Heal..

[16]  Duc Tien Dang Nguyen,et al.  HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy , 2019, Scientific Data.

[17]  Manolya Kavakli-Thorne,et al.  Applications of Generative Adversarial Networks (GANs): An Updated Review , 2019, Archives of Computational Methods in Engineering.

[18]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[19]  Thomas de Lange,et al.  Kvasir-SEG: A Segmented Polyp Dataset , 2019, MMM.

[20]  Noor Almaadeed,et al.  Image Inpainting: A Review , 2019, Neural Processing Letters.

[21]  Ulas Bagci,et al.  Quality assurance of computer-aided detection and diagnosis in colonoscopy. , 2019, Gastrointestinal endoscopy.

[22]  Yan Liu,et al.  Computer-aided diagnosis of colorectal polyps using linked color imaging colonoscopy to predict histology , 2019, Scientific Reports.

[23]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[24]  Tom White,et al.  Generative Adversarial Networks: An Overview , 2017, IEEE Signal Processing Magazine.

[25]  Sébastien Ourselin,et al.  Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations , 2017, DLMIA/ML-CDS@MICCAI.

[26]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[27]  Michael Riegler,et al.  KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection , 2017, MMSys.

[28]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.

[29]  Serge J. Belongie,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Michael Riegler,et al.  EIR — Efficient computer aided diagnosis framework for gastrointestinal endoscopies , 2016, 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI).

[32]  Nima Tajbakhsh,et al.  Automated Polyp Detection in Colonoscopy Videos Using Shape and Context Information , 2016, IEEE Transactions on Medical Imaging.

[33]  Fernando Vilariño,et al.  WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians , 2015, Comput. Medical Imaging Graph..

[34]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[37]  Aymeric Histace,et al.  Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer , 2014, International Journal of Computer Assisted Radiology and Surgery.

[38]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[39]  Marcin Polkowski,et al.  Quality indicators for colonoscopy and the risk of interval cancer. , 2010, The New England journal of medicine.

[40]  C. Le Berre,et al.  Application of Artificial Intelligence to Gastroenterology and Hepatology. , 2019, Gastroenterology.

[41]  G. Iddan,et al.  Wireless Capsule Endoscopy , 2013 .