Beware of diffusion models for synthesizing medical images - A comparison with GANs in terms of memorizing brain tumor images

Diffusion models were initially developed for text-to-image generation and are now being utilized to generate high quality synthetic images. Preceded by GANs, diffusion models have shown impressive results using various evaluation metrics. However, commonly used metrics such as FID and IS are not suitable for determining whether diffusion models are simply reproducing the training images. Here we train StyleGAN and diffusion models, using BRATS20 and BRATS21 datasets, to synthesize brain tumor images, and measure the correlation between the synthetic images and all training images. Our results show that diffusion models are much more likely to memorize the training images, especially for small datasets. Researchers should be careful when using diffusion models for medical imaging, if the final goal is to share the synthetic images.

[1]  Richard L. J. Qiu,et al.  2D medical image synthesis using transformer-based denoising diffusion probabilistic model , 2023, Physics in medicine and biology.

[2]  Florian Tramèr,et al.  Extracting Training Data from Diffusion Models , 2023, USENIX Security Symposium.

[3]  Jakob Nikolas Kather,et al.  Medical Diffusion: Denoising Diffusion Probabilistic Models for 3D Medical Image Generation , 2022, 2211.03364.

[4]  Puria Azadi Moghadam,et al.  A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Images , 2022, 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[5]  Ehsan Khodapanah Aghdam,et al.  Diffusion Models for Medical Image Analysis: A Comprehensive Survey , 2022, ArXiv.

[6]  A. Eklund,et al.  Does an ensemble of GANs lead to better performance when training segmentation networks with synthetic images? , 2022, ArXiv.

[7]  Zubair Shah,et al.  Spot the fake lungs: Generating Synthetic Medical Images using Neural Diffusion Models , 2022, AICS.

[8]  Florian Thamm,et al.  Generation of Anonymous Chest Radiographs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systems , 2022, 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI).

[9]  S. Ourselin,et al.  Brain Imaging Generation with Latent Diffusion Models , 2022, DGM4MICCAI@MICCAI.

[10]  Rebecka Jörnsten,et al.  On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity , 2022, NeurIPS.

[11]  David J. Fleet,et al.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding , 2022, NeurIPS.

[12]  Prafulla Dhariwal,et al.  Hierarchical Text-Conditional Image Generation with CLIP Latents , 2022, ArXiv.

[13]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Ali Borji,et al.  Pros and Cons of GAN Evaluation Measures: New Developments , 2021, Comput. Vis. Image Underst..

[15]  Prafulla Dhariwal,et al.  Diffusion Models Beat GANs on Image Synthesis , 2021, NeurIPS.

[16]  Alec Radford,et al.  Zero-Shot Text-to-Image Generation , 2021, ICML.

[17]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[18]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[19]  Paul Babyn,et al.  Generative Adversarial Network in Medical Imaging: A Review , 2018, Medical Image Anal..

[20]  Ali Borji,et al.  Pros and Cons of GAN Evaluation Measures , 2018, Comput. Vis. Image Underst..

[21]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[22]  Christos Davatzikos,et al.  Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features , 2017, Scientific Data.

[23]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[24]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[25]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[26]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.