CLIDiM: Contrastive Learning for Image Denoising in Microscopy

—Microscopy images often suffer from high levels of noise, which can hinder further analysis and interpretation. Content-aware image restoration (CARE) methods have been proposed to address this issue, but they often require large amounts of training data and suffer from over-fitting. To over- come these challenges, we propose a novel framework for few-shot microscopy image denoising. Our approach combines a generative adversarial network (GAN) trained via contrastive learning (CL) with two structure preserving loss terms – Struc- tural Similarity Index and Total Variation loss – to further improve the quality of the denoised images using little data. We demonstrate the effectiveness of our method on three well-known microscopy imaging datasets, and show that we can drastically reduce the amount of training data while retaining the quality of the denoising, thus alleviating the burden of acquiring paired data and enabling few-shot learning. The proposed framework can be easily extended to other image restoration tasks and has the potential to significantly advance the field of microscopy image analysis.

[1]  Walid El Shafai,et al.  Vision Transformers in Image Restoration: A Survey , 2023, Sensors.

[2]  F. Levet,et al.  MID3A: Microscopy Image Denoising meets Differentiable Data Augmentation , 2022, 2022 International Joint Conference on Neural Networks (IJCNN).

[3]  Syed Waqas Zamir,et al.  Restormer: Efficient Transformer for High-Resolution Image Restoration , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jianmin Bao,et al.  Uformer: A General U-Shaped Transformer for Image Restoration , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Se Young Chun,et al.  Rethinking Deep Image Prior for Denoising , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  Jong Chul Ye,et al.  Cycle-Free CycleGAN Using Invertible Generator for Unsupervised Low-Dose CT Denoising , 2021, IEEE Transactions on Computational Imaging.

[7]  Xuhui Huang,et al.  Robust Denoising of Cryo-EM Images via β-GAN , 2021 .

[8]  Yi Zhang,et al.  Noise-Powered Disentangled Representation for Unsupervised Speckle Reduction of Optical Coherence Tomography Images , 2020, IEEE Transactions on Medical Imaging.

[9]  Florian Jug,et al.  Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders , 2021, ICLR.

[10]  P. Milanfar,et al.  Removing Pixel Noises and Spatial Artifacts with Generative Diversity Denoising Methods , 2021, ArXiv.

[11]  Shuiwang Ji,et al.  Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising , 2020, NeurIPS.

[12]  Mingqin Chen,et al.  Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ce Liu,et al.  Supervised Contrastive Learning , 2020, NeurIPS.

[14]  Martin Weigert,et al.  Removing Structured Noise with Self-Supervised Blind-Spot Networks , 2020, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[15]  Sunil Agrawal,et al.  Image denoising review: From classical to state-of-the-art approaches , 2020, Inf. Fusion.

[16]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[17]  Philip H. S. Torr,et al.  Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Florian Jug,et al.  Probabilistic Noise2Void: Unsupervised Content-Aware Denoising , 2019, Frontiers in Computer Science.

[19]  Xianming Liu,et al.  Connecting Image Denoising and High-Level Vision Tasks via Deep Learning , 2018, IEEE Transactions on Image Processing.

[20]  Volker Tresp,et al.  Noise as Domain Shift: Denoising Medical Images by Unpaired Image Translation , 2019, DART/MIL3ID@MICCAI.

[21]  Xiaogang Wang,et al.  Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis , 2019, NeurIPS.

[22]  Fan Zhang,et al.  Brief review of image denoising techniques , 2019, Vis. Comput. Ind. Biomed. Art.

[23]  Jiliu Zhou,et al.  Simultaneous denoising and super-resolution of optical coherence tomography images based on generative adversarial network. , 2019, Optics express.

[24]  Taesung Park,et al.  Semantic Image Synthesis With Spatially-Adaptive Normalization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Loïc Royer,et al.  Noise2Self: Blind Denoising by Self-Supervision , 2019, ICML.

[26]  Tero Karras,et al.  High-Quality Self-Supervised Deep Image Denoising , 2019, NeurIPS.

[27]  Yide Zhang,et al.  A Poisson-Gaussian Denoising Dataset With Real Fluorescence Microscopy Images , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Florian Jug,et al.  Noise2Void - Learning Denoising From Single Noisy Images , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Radu Timofte,et al.  A Brief Review of Image Denoising Algorithms and Beyond , 2019, Inpainting and Denoising Challenges.

[30]  Ming Yang,et al.  Image Blind Denoising with Generative Adversarial Network Based Noise Modeling , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Jaakko Lehtinen,et al.  Noise2Noise: Learning Image Restoration without Clean Data , 2018, ICML.

[32]  Min Su,et al.  Generative adversarial networks as a tool to recover structural information from cryo-electron microscopy data , 2018, bioRxiv.

[33]  Loic A. Royer,et al.  Content-aware image restoration: pushing the limits of fluorescence microscopy , 2018, bioRxiv.

[34]  Eugene W. Myers,et al.  Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks , 2017, MICCAI.

[35]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[37]  Ole Winther,et al.  Autoencoding beyond pixels using a learned similarity metric , 2015, ICML.

[38]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[39]  Peyman Milanfar,et al.  A Tour of Modern Image Filtering: New Insights and Methods, Both Practical and Theoretical , 2013, IEEE Signal Processing Magazine.

[40]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[41]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[42]  Cedric Nishan Canagarajah,et al.  Structural Similarity-Based Object Tracking in Video Sequences , 2006, 2006 9th International Conference on Information Fusion.

[43]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[44]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[45]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[46]  Richard G. Baraniuk,et al.  Improved wavelet denoising via empirical Wiener filtering , 1997, Optics & Photonics.

[47]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .