Biological image analysis using deep learning-based methods: Literature review

Automatic processing large amount of microscopic images is important for medical and biological studies. Deep learning has demonstrated better performance than traditional machine learning methods for processing massive quantities of images; therefore, it has attracted increasing attention from the research and industry fields. This paper summarizes the latest progress of deep learning methods in biological microscopic image processing, including image classification, object detection, and image segmentation. Compared to the traditional machine learning methods, deep neural networks achieved better accuracy without tedious feature selection procedure. Obstacles of the biological image analysis with deep learning methods include limited training set and imperfect image quality. Viable solutions to these obstacles are discussed at the end of the paper. With this survey, we hope to provide a reference for the researchers conducting biological microscopic image processing.

[1]  Raghuveer Parthasarathy,et al.  Performance of convolutional neural networks for identification of bacteria in 3D microscopy datasets , 2018 .

[2]  Rajesh Krishna Balan,et al.  DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications , 2017, MobiSys.

[3]  Samuel J. Yang,et al.  In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images , 2018, Cell.

[4]  Ting Liu,et al.  Recent advances in convolutional neural networks , 2015, Pattern Recognit..

[5]  Tao Zhang,et al.  Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges , 2018, IEEE Signal Processing Magazine.

[6]  Mark J Schnitzer,et al.  In vivo optical microendoscopy for imaging cells lying deep within live tissue. , 2012, Cold Spring Harbor protocols.

[7]  Yi Li,et al.  R-FCN: Object Detection via Region-based Fully Convolutional Networks , 2016, NIPS.

[8]  Qiu Guan,et al.  Cancer Cells Detection in Phase-Contrast Microscopy Images Based on Faster R-CNN , 2016, 2016 9th International Symposium on Computational Intelligence and Design (ISCID).

[9]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[10]  Markus Reischl,et al.  Automated prior knowledge-based quantification of neuronal patterns in the spinal cord of zebrafish , 2014, Bioinform..

[11]  Ashutosh Kumar Singh,et al.  A deep learning framework to discern and count microscopic nematode eggs , 2018, Scientific Reports.

[12]  Teng Li,et al.  Facial Expression Recognition with Faster R-CNN , 2017 .

[13]  Aydogan Ozcan,et al.  Toward a Thinking Microscope: Deep Learning in Optical Microscopy and Image Reconstruction , 2018, Optics and Photonics News.

[14]  Xiao Zhang,et al.  Convolutional Neural Networks in Automatic Recognition of Trans-differentiated Neural Progenitor Cells under Bright-Field Microscopy , 2015, 2015 Fifth International Conference on Instrumentation and Measurement, Computer, Communication and Control (IMCCC).

[15]  P. Benfey,et al.  High-throughput phenotyping of multicellular organisms: finding the link between genotype and phenotype , 2011, Genome Biology.

[16]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[17]  Raphaël Marée,et al.  Automated processing of zebrafish imaging data: a survey. , 2013, Zebrafish.

[18]  Michael Tsang,et al.  Automated image‐based phenotypic analysis in zebrafish embryos , 2009, Developmental dynamics : an official publication of the American Association of Anatomists.

[19]  Philipp J. Keller,et al.  Reconstruction of Zebrafish Early Embryonic Development by Scanned Light Sheet Microscopy , 2008, Science.

[20]  Polina Golland,et al.  Voronoi-Based Segmentation of Cells on Image Manifolds , 2005, CVBIA.

[21]  D. LaBarbera,et al.  3D High-Content Screening of Organoids for Drug Discovery , 2017 .

[22]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[23]  Johannes E. Schindelin,et al.  Fiji: an open-source platform for biological-image analysis , 2012, Nature Methods.

[24]  Lutz Mädler,et al.  High content screening in zebrafish speeds up hazard ranking of transition metal oxide nanoparticles. , 2011, ACS nano.

[25]  Kevin W Eliceiri,et al.  NIH Image to ImageJ: 25 years of image analysis , 2012, Nature Methods.

[26]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[27]  Ronald M. Summers,et al.  Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique , 2016 .

[28]  Kevin W. Eliceiri,et al.  ImageJ2: ImageJ for the next generation of scientific image data , 2017, BMC Bioinformatics.

[29]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[30]  Jens Rittscher,et al.  Spatio-temporal cell cycle phase analysis using level sets and fast marching methods , 2009, Medical Image Anal..

[31]  Hang Li,et al.  Convolutional Neural Network Architectures for Matching Natural Language Sentences , 2014, NIPS.

[32]  Mart Saarma,et al.  A deep convolutional neural network approach for astrocyte detection , 2017 .

[33]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[34]  Kees-Jan van der Kolk,et al.  Deep learning image recognition enables efficient genome editing in zebrafish by automated injections , 2018 .

[35]  P. Gönczy,et al.  ASSET: A robust algorithm for the automated segmentation and standardization of early Caenorhabditis elegans embryos , 2010, Developmental dynamics : an official publication of the American Association of Anatomists.

[36]  Hui Huang,et al.  A portable high-resolution microscope based on combination of fiber-optic array and pre-amplification lens , 2018, Measurement.

[37]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[38]  R. Prevedel,et al.  Brain-wide 3D imaging of neuronal activity in Caenorhabditis elegans with sculpted light , 2013, Nature Methods.

[39]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Zi Wang,et al.  Deep reinforcement learning of cell movement in the early stage of C.elegans embryogenesis , 2018, Bioinform..

[41]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[42]  Ling Shao,et al.  Deep learning for automatic cell detection in wide-field microscopy zebrafish images , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[43]  Syed Muhammad Anwar,et al.  Deep Learning in Medical Image Analysis , 2017 .

[44]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Yann LeCun,et al.  Toward automatic phenotyping of developing embryos from videos , 2005, IEEE Transactions on Image Processing.

[46]  Aydogan Ozcan,et al.  Deep learning achieves super-resolution in fluorescence microscopy , 2018, bioRxiv.

[47]  Jieping Ye,et al.  Deep Model Based Transfer and Multi-Task Learning for Biological Image Analysis , 2015, IEEE Transactions on Big Data.

[48]  M. Gustafsson,et al.  Super-resolution 3D microscopy of live whole cells using structured illumination , 2011, Nature Methods.

[49]  Juho Kannala,et al.  Cell proposal network for microscopy image analysis , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[50]  K. I-K Wang,et al.  Automatic image analysis of zebrafish embryo development for lab-on-a-chip , 2012, 2012 19th International Conference on Mechatronics and Machine Vision in Practice (M2VIP).

[51]  Zhuowen Tu,et al.  Weakly supervised histopathology cancer image segmentation and classification , 2014, Medical Image Anal..

[52]  Douglas M. Hawkins,et al.  The Problem of Overfitting , 2004, J. Chem. Inf. Model..

[53]  O. Stegle,et al.  Deep learning for computational biology , 2016, Molecular systems biology.

[54]  David B. A. Epstein,et al.  MIMO-Net: A multi-input multi-output convolutional neural network for cell segmentation in fluorescence microscopy images , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[55]  I. A. Toker,et al.  WorMachine: machine learning-based phenotypic analysis tool for worms , 2017, BMC Biology.

[56]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Anne E Carpenter,et al.  Quantifying co-cultured cell phenotypes in high-throughput using pixel-based classification. , 2015, Methods.

[58]  Markus Reischl,et al.  Automated high-throughput mapping of promoter-enhancer interactions in zebrafish embryos , 2009, Nature Methods.

[59]  Yibo Zhang,et al.  Deep Learning Microscopy , 2017, ArXiv.

[60]  Eduardo D. Sontag,et al.  Rapid and accurate developmental stage recognition of C. elegans from high-throughput image data , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[61]  Raphaël Marée,et al.  A rich internet application for remote visualization and collaborative annotation of digital slides in histology and cytology , 2013, Diagnostic Pathology.

[62]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Qionghai Dai,et al.  Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning , 2018, 1805.05139.

[64]  Alberto Signoroni,et al.  Bacterial colony counting with Convolutional Neural Networks in Digital Microbiology Imaging , 2017, Pattern Recognit..

[65]  Jonathan Tompson,et al.  Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation , 2014, NIPS.

[66]  Leopold Parts,et al.  Accurate classification of protein subcellular localization from high throughput microscopy images using deep learning , 2016 .

[67]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[68]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[70]  Andrew Jones.  Segmenting Microarrays with Deep Neural Networks , 2015, bioRxiv.

[71]  F. Piano,et al.  A High-Resolution C. elegans Essential Gene Network Based on Phenotypic Profiling of a Complex Tissue , 2011, Cell.

[72]  J. Swoger,et al.  Light-sheet-based fluorescence microscopy for three-dimensional imaging of biological samples. , 2014, Cold Spring Harbor protocols.

[73]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  N. Jayalakshmi.  Cell Lineage Construction of Neural Progenitor Cells , 2014 .

[75]  Gerd Schneider,et al.  Three-dimensional cellular ultrastructure resolved by X-ray microscopy , 2010, Nature Methods.

[76]  Tao Xiang,et al.  Weakly-Supervised Image Annotation and Segmentation with Objects and Attributes , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Gilles Louppe,et al.  Collaborative analysis of multi-gigapixel imaging data using Cytomine , 2016, Bioinform..

[78]  Michael Unser,et al.  Advanced image processing for biology, and the Open Bio Image Alliance (OBIA) , 2013, 21st European Signal Processing Conference (EUSIPCO 2013).

[79]  Xiaogang Wang,et al.  Deep Convolutional Network Cascade for Facial Point Detection , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[80]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[81]  Yibo Zhang,et al.  PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning , 2018, Light: Science & Applications.

[82]  Raphaël Marée,et al.  Phenotype Classification of Zebrafish Embryos by Supervised Learning , 2012, PloS one.

[83]  Min Xu,et al.  Learn to segment single cells with deep distance estimator and deep cell detector , 2018, Comput. Biol. Medicine.

[84]  Haiying Liu,et al.  A Generic Framework for Non-rigid Registration Based on Non-uniform Multi-level Free-Form Deformations , 2001, MICCAI.

[85]  Eduardo D. Sontag,et al.  DevStaR: High-Throughput Quantification of C. elegans Developmental Stages , 2013, IEEE Transactions on Medical Imaging.

[86]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[87]  Raphaël Marée,et al.  Towards generic image classification using tree-based learning: An extensive empirical study , 2016, Pattern Recognit. Lett..

[88]  Gwénolé Quellec,et al.  Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks , 2018, Medical Image Anal..

[89]  Andrew Zisserman,et al.  Convolutional Two-Stream Network Fusion for Video Action Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[90]  Shuiwang Ji,et al.  Deep Learning Segmentation of Optical Microscopy Images Improves 3-D Neuron Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[91]  Vijay S. Pande,et al.  Using Deep Learning for Segmentation and Counting within Microscopy Data , 2018, ArXiv.

[92]  B. S. Manjunath,et al.  Biological imaging software tools , 2012, Nature Methods.

[93]  Yolanda T. Chong,et al.  Automated analysis of high‐content microscopy data with deep learning , 2017, Molecular systems biology.

[94]  Rita Strack.  Deep learning advances super-resolution imaging , 2018, Nature Methods.

[95]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[96]  Robert Rallo,et al.  Automated Phenotype Recognition for Zebrafish Embryo Based In Vivo High Throughput Toxicity Screening of Engineered Nano-Materials , 2012, PloS one.

[97]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[98]  Brendan J. Frey,et al.  Classifying and segmenting microscopy images with deep multiple instance learning , 2015, Bioinform..

[99]  Yibo Zhang,et al.  Deep learning enhanced mobile-phone microscopy , 2017, ACS Photonics.

[100]  Chandra Kambhamettu,et al.  DeepXScope: Segmenting Microscopy Images with a Deep Neural Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[101]  J. Bobe.  Egg quality in fish: Present and future challenges , 2015 .

[102]  A. Verbeke,et al.  Automatic standardisation of a zebrafish embryo image database , 2009, 2009 9th International Conference on Information Technology and Applications in Biomedicine.

[103]  Thomas Brox,et al.  ViBE-Z: a framework for 3D virtual colocalization analysis in zebrafish larval brains , 2012, Nature Methods.

[104]  Byunghan Lee,et al.  Deep learning in bioinformatics , 2016, Briefings Bioinform..

[105]  Christoph Sommer,et al.  Machine learning in cell biology – teaching computers to recognize phenotypes , 2013, Journal of Cell Science.

[106]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[107]  Atsuto Maki,et al.  Factors of Transferability for a Generic ConvNet Representation , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.