Image generation by GAN and style transfer for agar plate image segmentation

BACKGROUND AND OBJECTIVES Deep learning models and specifically Convolutional Neural Networks (CNNs) are becoming the leading approach in many computer vision tasks, including medical image analysis. Nevertheless, the CNN training usually requires large sets of supervised data, which are often difficult and expensive to obtain in the medical field. To address the lack of annotated images, image generation is a promising method, which is becoming increasingly popular in the computer vision community. In this paper, we present a new approach to the semantic segmentation of bacterial colonies in agar plate images, based on deep learning and synthetic image generation, to increase the training set size. Indeed, semantic segmentation of bacterial colony is the basis for infection recognition and bacterial counting in Petri plate analysis. METHODS A convolutional neural network (CNN) is used to separate the bacterial colonies from the background. To face the lack of annotated images, a novel engine is designed - which exploits a generative adversarial network to capture the typical distribution of the bacterial colonies on agar plates - to generate synthetic data. Then, bacterial colony patches are superimposed on existing background images, taking into account both the local appearance of the background and the intrinsic opacity of the bacterial colonies, and a style transfer algorithm is used for further improve visual realism. RESULTS The proposed deep learning approach has been tested on the only public dataset available with pixel-level annotations for bacterial colony semantic segmentation in agar plates. The role of including synthetic data in the training of a segmentation CNN has been evaluated, showing how comparable performances can be obtained with respect to the use of real images. Qualitative results are also reported for a second public dataset in which the segmentation annotations are not provided. CONCLUSIONS The use of a small set of real data, together with synthetic images, allows obtaining comparable results with respect to using a complete set of real images. Therefore, the proposed synthetic data generator is able to address the scarcity of biomedical data and provides a scalable and cheap alternative to human ground-truth supervision.

[1]  Andrea Garzelli,et al.  ABLE: An Automated Bacterial Load Estimator for the Urinoculture Screening , 2016, ICPRAM.

[2]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[3]  Alberto Signoroni,et al.  Automatic hemolysis identification on aligned dual-lighting images of cultured blood agar plates , 2018, Comput. Methods Programs Biomed..

[4]  J. Folch-Mallol,et al.  COVASIAM: an Image Analysis Method That Allows Detection of Confluent Microbial Colonies and Colonies of Various Sizes for Automated Counting , 1998, Applied and Environmental Microbiology.

[5]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[6]  Jeffrey L. Gunter,et al.  Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks , 2018, SASHIMI@MICCAI.

[7]  Alessandro Mecocci,et al.  Automatic Image Analysis and Classification for Urinary Bacteria Infection Screening , 2015, ICIAP.

[8]  Markus Schoeler,et al.  Semantic Pose Using Deep Networks Trained on Synthetic RGB-D , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  N. E. Alexander,et al.  Automatic Counting of Bacterial Cultures- A New Machine , 1958 .

[10]  Franco Scarselli,et al.  A Deep Learning Approach to Bacterial Colony Segmentation , 2018, ICANN.

[11]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[12]  Dong Han,et al.  Adaptive Ideal Image Reconstruction for Bacteria Colony Detection , 2012 .

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[15]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Hayit Greenspan,et al.  Synthetic data augmentation using GAN for improved liver lesion classification , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[19]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[20]  Antonio M. López,et al.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  H P MANSBERG,et al.  Automatic particle and bacterial colony counter. , 1957, Science.

[22]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[24]  Ankush Gupta,et al.  Synthetic Data for Text Localisation in Natural Images , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Andrea Vedaldi,et al.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images , 2016, ICML.

[26]  George Papandreou,et al.  Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[27]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Franco Scarselli,et al.  Generating Bounding Box Supervision for Semantic Segmentation with Deep Learning , 2018, ANNPR.

[29]  Vladlen Koltun,et al.  Playing for Data: Ground Truth from Computer Games , 2016, ECCV.

[30]  Anil A. Bharath,et al.  Adversarial Training for Sketch Retrieval , 2016, ECCV Workshops.

[31]  Cordelia Schmid,et al.  P-CNN: Pose-Based CNN Features for Action Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[32]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[33]  Alberto Signoroni,et al.  Multistage classification for bacterial colonies recognition on solid agar images , 2014, 2014 IEEE International Conference on Imaging Systems and Techniques (IST) Proceedings.

[34]  Quentin Geissmann,et al.  OpenCFU, a New Free and Open-Source Software to Count Cell Colonies and Other Circular Objects , 2012, PloS one.

[35]  Alberto Signoroni,et al.  Bacterial colony counting with Convolutional Neural Networks in Digital Microbiology Imaging , 2017, Pattern Recognit..

[36]  Wei-bang Chen,et al.  An Automated Bacterial Colony Counting System , 2008, 2008 IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (sutc 2008).

[37]  Urs Brugger,et al.  Automated Counting of Bacterial Colony Forming Units on Agar Plates , 2012, PloS one.

[38]  A Pal,et al.  Bacterial colony counting using distance transform. , 1995, International journal of bio-medical computing.

[39]  Cordelia Schmid,et al.  How good is my GAN? , 2018, ECCV.

[40]  Takeo Kanade,et al.  Learning scene-specific pedestrian detectors without real data , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  R. G. Mcguire,et al.  Reporting of Objective Color Measurements , 1992 .

[42]  Piernicola Oliva,et al.  Automatic cell colony counting by region-growing approach , 2007 .

[43]  Jonathon Shlens,et al.  A Learned Representation For Artistic Style , 2016, ICLR.

[44]  Andrew Zisserman,et al.  Reading Text in the Wild with Convolutional Neural Networks , 2014, International Journal of Computer Vision.

[45]  Xueting Li,et al.  A Closed-form Solution to Photorealistic Image Stylization , 2018, ECCV.

[46]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[47]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Mario Lucic,et al.  Are GANs Created Equal? A Large-Scale Study , 2017, NeurIPS.

[49]  Matthieu Cord,et al.  WILDCAT: Weakly Supervised Learning of Deep ConvNets for Image Classification, Pointwise Localization and Segmentation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Alessandro Mecocci,et al.  Extraction of High Level Visual Features for the Automatic Recognition of UTIs , 2016, WILF.

[51]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[52]  Ming-Hsuan Yang,et al.  Universal Style Transfer via Feature Transforms , 2017, NIPS.

[53]  Alexei A. Efros,et al.  Generative Visual Manipulation on the Natural Image Manifold , 2016, ECCV.

[54]  Juergen Gall,et al.  Adaptation of Synthetic Data for Coarse-to-Fine Viewpoint Refinement , 2015, BMVC.

[55]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[56]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Dumitru Erhan,et al.  Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Alessandro Mecocci,et al.  Automatic Image Classification for the Urinoculture Screening , 2015, KES-IDT.

[60]  Andrea Garzelli,et al.  Automatic image classification for the urinoculture screening , 2016, Comput. Biol. Medicine.

[61]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[62]  David Vázquez,et al.  Learning appearance in virtual scenarios for pedestrian detection , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.