Learning to Restore ssTEM Images from Deformation and Corruption

Serial section transmission electron microscopy (ssTEM) plays an important role in biological research. Due to the imperfect sample preparation, however, ssTEM images suffer from inevitable artifacts that pose huge challenges for the subsequent analysis and visualization. In this paper, we propose a novel strategy for modeling the main type of degradation, i.e., Support Film Folds (SFF), by characterizing this degradation process as a combination of content deformation and corruption. Relying on that, we then synthesize a sufficient amount of paired samples (degraded/groundtruth), which enables the training of a tailored deep restoration network. To the best of our knowledge, this is the first learning-based framework for ssTEM image restoration. Experiments on both synthetic and real test data demonstrate the superior performance of our proposed method over existing solutions, in terms of both image restoration quality and neuron segmentation accuracy.

[1]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[2]  Baining Guo,et al.  Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Jan Funke.  Automatic Neuron Reconstruction from Anisotropic Electron Microscopy Volumes , 2014 .

[4]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Separable Convolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Chandan Singh,et al.  Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Arthur W. Wetzel,et al.  Network anatomy and in vivo physiology of visual cortical neurons , 2011, Nature.

[10]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[11]  Won-Ki Jeong,et al.  Removing Imaging Artifacts in Electron Microscopy using an Asymmetrically Cyclic Adversarial Network without Paired Training Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[12]  J. Bourne,et al.  Uniform Serial Sectioning for Transmission Electron Microscopy , 2006, The Journal of Neuroscience.

[13]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Convolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[15]  Faisal Z. Qureshi,et al.  EdgeConnect: Structure Guided Image Inpainting using Edge Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[16]  Kai-Kuang Ma,et al.  A new diamond search algorithm for fast block-matching motion estimation , 2000, IEEE Trans. Image Process..

[17]  H. Sebastian Seung,et al.  Caesar: Segment-Wise Alignment Method for Solving Discontinuous Deformations , 2020, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[18]  Dmitri B. Chklovskii,et al.  Strip the Stripes: Artifact Detection and Removal for Scanning Electron Microscopy Imaging , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[19]  Stephen J. Smith,et al.  High-contrast en bloc staining of neuronal tissue for field emission scanning electron microscopy , 2012, Nature Protocols.

[20]  Feng Li,et al.  Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment , 2019 .

[21]  J Roels,et al.  An overview of state‐of‐the‐art image restoration in electron microscopy , 2018, Journal of microscopy.

[22]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.

[23]  Jianbo Shi,et al.  Machine Learning of Hierarchical Clustering to Segment 2D and 3D Images , 2013, PloS one.

[24]  Joachim M. Buhmann,et al.  Crowdsourcing the creation of image segmentation algorithms for connectomics , 2015, Front. Neuroanat..

[25]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[26]  S. Maraghechi,et al.  Correction of Scanning Electron Microscope Imaging Artifacts in a Novel Digital Image Correlation Framework , 2019, Experimental Mechanics.

[27]  G. Knott,et al.  Serial Section Scanning Electron Microscopy of Adult Brain Tissue Using Focused Ion Beam Milling , 2008, The Journal of Neuroscience.

[28]  Ting-Chun Wang,et al.  Image Inpainting for Irregular Holes Using Partial Convolutions , 2018, ECCV.

[29]  W. Denk,et al.  Serial Block-Face Scanning Electron Microscopy to Reconstruct Three-Dimensional Tissue Nanostructure , 2004, PLoS biology.

[30]  Eric T. Trautman,et al.  A Complete Electron Microscopy Volume of the Brain of Adult Drosophila melanogaster , 2017, Cell.

[31]  Claus Ropers,et al.  Quantum coherent optical phase modulation in an ultrafast transmission electron microscope , 2015, Nature.

[32]  Isabella Haberbosch,et al.  Software tools for automated transmission electron microscopy , 2018, Nature Methods.