Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder

We present an atrous convolutional encoder-decoder trained to denoise electron micrographs. It consists of a modified Xception backbone, atrous convoltional spatial pyramid pooling module and a multi-stage decoder. Our neural network was trained end-to-end using 512  ×  512 micrographs created from a large dataset of high-dose ( > 2500 counts per pixel) micrographs with added Poisson noise to emulate low-dose ( ≪  300 counts per pixel) data. It was then fine-tuned for high dose data (200-2500 counts per pixel). Its performance is benchmarked against bilateral, Gaussian, median, total variation, wavelet, and Wiener restoration methods with their default parameters. Our network outperforms their best mean squared error and structural similarity index performances by 24.6% and 9.6% for low doses and by 43.7% and 5.5% for high doses. In both cases, our network's mean squared error has the lowest variance. Source code and links to our high-quality dataset and pre-trained models are available at https://github.com/Jeffrey-Ede/Electron-Micrograph-Denoiser.

[1]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  H. Kile,et al.  Bandwidth Selection in Kernel Density Estimation , 2010 .

[3]  Xuanqin Mou,et al.  Low-Dose CT Image Denoising Using a Generative Adversarial Network With Wasserstein Distance and Perceptual Loss , 2017, IEEE Transactions on Medical Imaging.

[4]  Christoph T Koch,et al.  Towards full-resolution inline electron holography. , 2014, Micron.

[5]  Woohyung Lim,et al.  Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: Automatic construction of onychomycosis datasets by region-based convolutional deep neural network , 2018, PloS one.

[6]  D. Narmadha,et al.  A Survey on Image Denoising Techniques , 2012 .

[7]  T. Shintake,et al.  Development of a SEM-based low-energy in-line electron holography microscope for individual particle imaging. , 2018, Ultramicroscopy.

[8]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[9]  Claus Ropers,et al.  Ultrafast transmission electron microscopy using a laser-driven field emitter: Femtosecond resolution with a high coherence electron beam. , 2017, Ultramicroscopy.

[10]  Vincent Dumoulin,et al.  Deconvolution and Checkerboard Artifacts , 2016 .

[11]  Max Tegmark,et al.  Why Does Deep and Cheap Learning Work So Well? , 2016, Journal of Statistical Physics.

[12]  M. Monthioux,et al.  Development of TEM and SEM high brightness electron guns using cold-field emission from a carbon nanotip. , 2015, Ultramicroscopy.

[13]  Tamir Gonen,et al.  Analysis of global and site-specific radiation damage in cryo-EM , 2018, bioRxiv.

[14]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[15]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Tim Salimans,et al.  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks , 2016, NIPS.

[17]  Eva L. Dyer,et al.  Low-dose x-ray tomography through a deep convolutional neural network , 2018, Scientific Reports.

[18]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[19]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[21]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[22]  A. Krizhevsky.  Convolutional Deep Belief Networks on CIFAR-10 , 2010 .

[23]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[24]  Geoffrey Zweig,et al.  Achieving Human Parity in Conversational Speech Recognition , 2016, ArXiv.

[25]  Sergey Ioffe,et al.  Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models , 2017, NIPS.

[26]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[27]  J. Biskupek,et al.  Chromatic Aberration Correction for Atomic Resolution TEM Imaging from 20 to 80 kV. , 2016, Physical review letters.

[28]  Qin Zhang,et al.  CRYO-ELECTRON MICROSCOPY DATA DENOISING BASED ON THE GENERALIZED DIGITIZED TOTAL VARIATION METHOD. , 2010, Far east journal of applied mathematics.

[29]  Samuel J. Yang,et al.  In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images , 2018, Cell.

[30]  Alexander M. Bronstein,et al.  Deep Convolutional Denoising of Low-Light Images , 2017, ArXiv.

[31]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[32]  Dong Yu,et al.  Single-channel mixed speech recognition using deep neural networks , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[33]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[34]  Martin Vetterli,et al.  Adaptive wavelet thresholding for image denoising and compression , 2000, IEEE Trans. Image Process..

[35]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[36]  Pascal Getreuer,et al.  Rudin-Osher-Fatemi Total Variation Denoising using Split Bregman , 2012, Image Process. Line.

[37]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[38]  S. Pennycook.  The impact of STEM aberration correction on materials science. , 2017, Ultramicroscopy.

[39]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[40]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[41]  Karel J. Zuiderveld,et al.  Contrast Limited Adaptive Histogram Equalization , 1994, Graphics Gems.

[42]  W Xu,et al.  A deep convolutional neural network to analyze position averaged convergent beam electron diffraction patterns. , 2017, Ultramicroscopy.

[43]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[44]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[45]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[46]  Ilya Kostrikov,et al.  PlaNet - Photo Geolocation with Convolutional Neural Networks , 2016, ECCV.

[47]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[48]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[49]  L. Strüder,et al.  Rapid low dose electron tomography using a direct electron detection camera , 2015, Scientific Reports.

[50]  John Tran,et al.  cuDNN: Efficient Primitives for Deep Learning , 2014, ArXiv.

[51]  Razvan Pascanu,et al.  A simple neural network module for relational reasoning , 2017, NIPS.

[52]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[53]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Rosemarie Velik,et al.  Discrete Fourier Transform Computation Using Neural Networks , 2008, 2008 International Conference on Computational Intelligence and Security.

[55]  Emmanuelle Gouillart,et al.  scikit-image: image processing in Python , 2014, PeerJ.

[56]  Rob J. Hyndman,et al.  Bandwidth selection for kernel conditional density estimation , 2001 .

[57]  Samy Bengio,et al.  Revisiting Distributed Synchronous SGD , 2016, ArXiv.

[58]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[59]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[60]  Francesco Visin,et al.  A guide to convolution arithmetic for deep learning , 2016, ArXiv.

[61]  Veit Elser,et al.  Electron ptychography of 2D materials to deep sub-ångström resolution , 2018, Nature.

[62]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[63]  Tao Xiang,et al.  Sketch-a-Net: A Deep Neural Network that Beats Humans , 2017, International Journal of Computer Vision.

[64]  N. Ichise,et al.  Practical method for noise removal in scanning electron microscopy , 2006 .

[65]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[66]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Joshua B. Tenenbaum,et al.  Beating the World's Best at Super Smash Bros. with Deep Reinforcement Learning , 2017, ArXiv.

[68]  M. Kosinski,et al.  Deep Neural Networks Are More Accurate Than Humans at Detecting Sexual Orientation From Facial Images , 2018, Journal of personality and social psychology.

[69]  A. Tonomura,et al.  Aberration corrected 1.2-MV cold field-emission transmission electron microscope with a sub-50-pm resolution , 2015 .

[70]  Daniel Cremers,et al.  Regularization for Deep Learning: A Taxonomy , 2017, ArXiv.

[71]  H. S. Kushwaha,et al.  De-noising Filters for TEM (Transmission Electron Microscopy) Image of Nanomaterials , 2012, 2012 Second International Conference on Advanced Computing & Communication Technologies.

[72]  Yanan Zhu,et al.  A deep convolutional neural network approach to single-particle recognition in cryo-electron microscopy , 2016, BMC Bioinformatics.

[73]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[74]  H. Sebastian Seung,et al.  Superhuman Accuracy on the SNEMI3D Connectomics Challenge , 2017, ArXiv.

[75]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[76]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[77]  Guillaume Lample,et al.  Playing FPS Games with Deep Reinforcement Learning , 2016, AAAI.

[78]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.

[79]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[80]  Yu-Bin Yang,et al.  Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections , 2016, ArXiv.