CleftNet: Augmented Deep Learning for Synaptic Cleft Detection From Brain Electron Microscopy

Detecting synaptic clefts is a crucial step to investigate the biological function of synapses. The volume electron microscopy (EM) allows the identification of synaptic clefts by photoing EM images with high resolution and fine details. Machine learning approaches have been employed to automatically predict synaptic clefts from EM images. In this work, we propose a novel and augmented deep learning model, known as CleftNet, for improving synaptic cleft detection from brain EM images. We first propose two novel network components, known as the feature augmentor and the label augmentor, for augmenting features and labels to improve cleft representations. The feature augmentor can fuse global information from inputs and learn common morphological patterns in clefts, leading to augmented cleft features. In addition, it can generate outputs with varying dimensions, making it flexible to be integrated in any deep network. The proposed label augmentor augments the label of each voxel from a value to a vector, which contains both the segmentation label and boundary label. This allows the network to learn important shape information and to produce more informative cleft representations. Based on the proposed feature augmentor and label augmentor, We build the CleftNet as a U-Net like network. The effectiveness of our methods is evaluated on both online and offline tasks. Our CleftNet currently ranks #1 on the online task of the CREMI open challenge. In addition, both quantitative and qualitative results in the offline tasks show that our method outperforms the baseline approaches significantly.

[1]  G. Urban,et al.  Automated synaptic connectivity inference for volume electron microscopy , 2017, Nature Methods.

[2]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[3]  Hanchuan Peng,et al.  Deep models for brain EM image segmentation: novel insights and improved performance , 2016, Bioinform..

[4]  Qingjie Liu,et al.  Road Extraction by Deep Residual U-Net , 2017, IEEE Geoscience and Remote Sensing Letters.

[5]  Sanja Fidler,et al.  Gated-SCNN: Gated Shape CNNs for Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  Tianwei Ni,et al.  Elastic Boundary Projection for 3D Medical Image Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ziv Yaniv,et al.  A Distance Map Regularized CNN for Cardiac Cine MR Image Segmentation , 2019, Medical physics.

[8]  Septimiu E. Salcudean,et al.  Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks , 2019, IEEE Transactions on Medical Imaging.

[9]  Shuiwang Ji,et al.  Global Voxel Transformer Networks for Augmented Microscopy , 2021, Nat. Mach. Intell..

[10]  Gunilla Borgefors,et al.  Distance transformations in digital images , 1986, Comput. Vis. Graph. Image Process..

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Shuiwang Ji,et al.  Global Pixel Transformers for Virtual Staining of Microscopy Images , 2019, IEEE Transactions on Medical Imaging.

[13]  Kaiming He,et al.  Improved Baselines with Momentum Contrastive Learning , 2020, ArXiv.

[14]  Samuel J. Yang,et al.  In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images , 2018, Cell.

[15]  Hao Yuan,et al.  Learning Local and Global Multi-context Representations for Document Classification , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[16]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[18]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[19]  Seyed-Ahmad Ahmadi,et al.  Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields , 2016, MICCAI.

[20]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Stephan Saalfeld,et al.  Synaptic Cleft Segmentation in Non-Isotropic Volume Electron Microscopy of the Complete Drosophila Brain , 2018, MICCAI.

[22]  Shuiwang Ji,et al.  Deep Learning Segmentation of Optical Microscopy Images Improves 3-D Neuron Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[23]  Shuiwang Ji,et al.  Residual Deconvolutional Networks for Brain Electron Microscopy Image Segmentation , 2017, IEEE Transactions on Medical Imaging.

[24]  Matthias Bethge,et al.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness , 2018, ICLR.

[25]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[26]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[27]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Sébastien Ourselin,et al.  Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning , 2017, IEEE Transactions on Medical Imaging.

[29]  Suyash P. Awate,et al.  Computer Vision, Graphics, and Image Processing , 2016, Lecture Notes in Computer Science.

[30]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[31]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Loïc Le Folgoc,et al.  Attention U-Net: Learning Where to Look for the Pancreas , 2018, ArXiv.

[33]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[34]  Yong Yin,et al.  Shape-Aware Organ Segmentation by Predicting Signed Distance Maps , 2019, AAAI.

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Wei-Chung Allen Lee,et al.  Automatic Detection of Synaptic Partners in a Whole-Brain Drosophila EM Dataset , 2019, bioRxiv.

[37]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[38]  Pascal Fua,et al.  Learning Context Cues for Synapse Segmentation , 2013, IEEE Transactions on Medical Imaging.

[39]  Casey M. Schneider-Mizell,et al.  Binary and analog variation of synapses between cortical pyramidal neurons , 2019, bioRxiv.

[40]  Fred A. Hamprecht,et al.  Automated Detection and Segmentation of Synaptic Contacts in Nearly Isotropic Serial Electron Microscopy Images , 2011, PloS one.

[41]  Dinggang Shen,et al.  Voxel Deconvolutional Networks for 3D Brain Image Labeling , 2018, KDD.

[42]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[43]  Liu,et al.  MitoEM Dataset: Large-Scale 3D Mitochondria Instance Segmentation from EM Images , 2020, MICCAI.

[44]  H. Sebastian Seung,et al.  Convolutional nets for reconstructing neural circuits from brain images acquired by serial section electron microscopy , 2019, Current Opinion in Neurobiology.

[45]  Bian Wu,et al.  DeepEM3D: approaching human‐level performance on 3D anisotropic EM image segmentation , 2017, Bioinform..

[46]  Eric T. Trautman,et al.  A Complete Electron Microscopy Volume of the Brain of Adult Drosophila melanogaster , 2017, Cell.

[47]  Dinggang Shen,et al.  Non-Local U-Nets for Biomedical Image Segmentation , 2020, AAAI.