mEMbrain: an interactive deep learning MATLAB tool for connectomic segmentation on commodity desktops

Connectomics is fundamental in propelling our understanding of the nervous system's organization, unearthing cells and wiring diagrams reconstructed from volume electron microscopy (EM) datasets. Such reconstructions, on the one hand, have benefited from ever more precise automatic segmentation methods, which leverage sophisticated deep learning architectures and advanced machine learning algorithms. On the other hand, the field of neuroscience at large, and of image processing in particular, has manifested a need for user-friendly and open source tools which enable the community to carry out advanced analyses. In line with this second vein, here we propose mEMbrain, an interactive MATLAB-based software which wraps algorithms and functions that enable labeling and segmentation of electron microscopy datasets in a user-friendly user interface compatible with Linux and Windows. Through its integration as an API to the volume annotation and segmentation tool VAST, mEMbrain encompasses functions for ground truth generation, image preprocessing, training of deep neural networks, and on-the-fly predictions for proofreading and evaluation. The final goals of our tool are to expedite manual labeling efforts and to harness MATLAB users with an array of semi-automatic approaches for instance segmentation. We tested our tool on a variety of datasets that span different species at various scales, regions of the nervous system and developmental stages. To further expedite research in connectomics, we provide an EM resource of ground truth annotation from four different animals and five datasets, amounting to around 180 h of expert annotations, yielding more than 1.2 GB of annotated EM images. In addition, we provide a set of four pre-trained networks for said datasets. All tools are available from https://lichtman.rc.fas.harvard.edu/mEMbrain/. With our software, our hope is to provide a solution for lab-based neural reconstructions which does not require coding by the user, thus paving the way to affordable connectomics.

[1]  J. Lichtman,et al.  Immersion Fixation and Staining of Multicubic Millimeter Volumes for Electron Microscopy–Based Connectomics of Human Brain Biopsies , 2023, Biological Psychiatry.

[2]  Heather G. Patsolic,et al.  The connectome of an insect brain , 2022, bioRxiv.

[3]  David Grant Colburn Hildebrand,et al.  Structured cerebellar connectivity supports resilient pattern separation , 2022, Nature.

[4]  B. Hochner,et al.  Connectomics of the Octopus vulgaris vertical lobe provides insight into conserved and novel principles of a memory acquisition network , 2022, bioRxiv.

[5]  Brett J. Graham,et al.  Synaptic wiring motifs in posterior parietal cortex support decision-making , 2022, bioRxiv.

[6]  Chris S. Jordan,et al.  Reconstruction of neocortex: Organelles, compartments, cells, circuits, and activity , 2022, Cell.

[7]  M. Zhen,et al.  Structural Analysis of the Caenorhabditis elegans Dauer Larval Anterior Sensilla by Focused Ion Beam-Scanning Electron Microscopy , 2021, Frontiers in Neuroanatomy.

[8]  R. W. Draft,et al.  Neuromuscular connectomes across development reveal synaptic ordering rules , 2021, bioRxiv.

[9]  Peter H. Li,et al.  A connectomic study of a petascale fragment of human cerebral cortex , 2021, bioRxiv.

[10]  Fabian J Theis,et al.  Cellular connectomes as arbiters of local circuit models in the cerebral cortex , 2021, Nature Communications.

[11]  James A. Gagnon,et al.  Functional and ultrastructural analysis of reafferent mechanosensation in larval zebrafish , 2021, Current Biology.

[12]  Frank Hutter,et al.  TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  M. Dickinson,et al.  Transforming representations of movement from body- to world-centric space , 2020, Nature.

[14]  Aljoscha Nern,et al.  The connectome of the adult Drosophila mushroom body provides insights into function , 2020, eLife.

[15]  Marius Pachitariu,et al.  Cellpose: a generalist algorithm for cellular segmentation , 2020, Nature Methods.

[16]  Bruce R. Rosen,et al.  The Mind of a Mouse , 2020, Cell.

[17]  G. Jékely,et al.  Whole-animal connectome and cell-type complement of the three-segmented Platynereis dumerilii larva , 2020, bioRxiv.

[18]  Matthew J. Johnson,et al.  Revealing the structure of pharmacobehavioral space through Motion Sequencing , 2020, Nature Neuroscience.

[19]  Daniel R. Berger,et al.  Connectomes across development reveal principles of brain maturation in C. elegans , 2020, bioRxiv.

[20]  Feng Li,et al.  A connectome and analysis of the adult Drosophila central brain , 2020, bioRxiv.

[21]  Marius Pachitariu,et al.  Cellpose: a generalist algorithm for cellular segmentation , 2020, Nature Methods.

[22]  J. Lichtman,et al.  Developmental Rewiring between Cerebellar Climbing Fibers and Purkinje Cells Begins with Positive Feedback Synapse Addition , 2019, Cell reports.

[23]  Quoc V. Le,et al.  Randaugment: Practical automated data augmentation with a reduced search space , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Fred A. Hamprecht,et al.  ilastik: interactive machine learning for (bio)image analysis , 2019, Nature Methods.

[25]  Quoc V. Le,et al.  AutoAugment: Learning Augmentation Strategies From Data , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Shin Ishii,et al.  UNI-EM: An Environment for Deep Neural Network-Based Automated Segmentation of Neuronal Electron Microscopic Images , 2019, Scientific Reports.

[27]  Hayk Saribekyan,et al.  Cross-Classification Clustering: An Efficient Multi-Object Tracking Technique for 3-D Instance Segmentation in Connectomics , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Ting Zhao,et al.  NeuTu: Software for Collaborative, Large-Scale, Segmentation-Based Connectome Reconstruction , 2018, Front. Neural Circuits.

[29]  Yvan Saeys,et al.  Domain Adaptive Segmentation In Volume Electron Microscopy Imaging , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[30]  H. Sebastian Seung,et al.  VAST (Volume Annotation and Segmentation Tool): Efficient Manual and Semi-Automatic Labeling of Large 3D Image Stacks , 2018, Front. Neural Circuits.

[31]  Kevin M. Cury,et al.  DeepLabCut: markerless pose estimation of user-defined body parts with deep learning , 2018, Nature Neuroscience.

[32]  William C. Smith,et al.  Photoreceptor specialization and the visuomotor repertoire of the primitive chordate Ciona , 2018, Journal of Experimental Biology.

[33]  Larry Lindsey,et al.  High-precision automated reconstruction of neurons with flood-filling networks , 2017, Nature Methods.

[34]  Philipp Otto,et al.  webKnossos: efficient online 3D data annotation for connectomics , 2017, Nature Methods.

[35]  Eric T. Trautman,et al.  A Complete Electron Microscopy Volume of the Brain of Adult Drosophila melanogaster , 2017, Cell.

[36]  G. Jékely,et al.  Ciliomotor circuitry underlying whole-body coordination of ciliary activity in the Platynereis larva , 2017, bioRxiv.

[37]  B. Hochner,et al.  The Vertical Lobe of Cephalopods—A Brain Structure Ideal for Exploring the Mechanisms of Complex Forms of Learning and Memory , 2017 .

[38]  Kenneth D. Harris,et al.  Fast and accurate spike sorting of high-channel count probes with KiloSort , 2016, NIPS.

[39]  Hayk Saribekyan,et al.  A Multi-Pass Approach to Large-Scale Connectomics , 2016, ArXiv.

[40]  Mario Dipoppa,et al.  Suite2p: beyond 10,000 neurons with standard two-photon microscopy , 2016, bioRxiv.

[41]  Brett J. Graham,et al.  Anatomy and function of an excitatory network in the visual cortex , 2016, Nature.

[42]  Moritz Helmstaedter,et al.  SegEM: Efficient Image Analysis for High-Resolution Connectomics , 2015, Neuron.

[43]  William R. Gray Roncal,et al.  Saturated Reconstruction of a Volume of Neocortex , 2015, Cell.

[44]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[45]  G. Knott,et al.  Ultrastructurally-smooth thick partitioning and volume stitching for larger-scale connectomics , 2015, Nature Methods.

[46]  Stephen D. Larson,et al.  OpenWorm: an open-science approach to modeling Caenorhabditis elegans , 2014, Front. Comput. Neurosci..

[47]  Nir Shavit,et al.  The big data challenges of connectomics , 2014, Nature Neuroscience.

[48]  David Grant Colburn Hildebrand,et al.  Imaging ATUM ultrathin section libraries with WaferMapper: a multi-scale approach to EM reconstruction of neural circuits , 2014, Front. Neural Circuits.

[49]  Srinivas C. Turaga,et al.  Connectomic reconstruction of the inner plexiform layer in the mouse retina , 2013, Nature.

[50]  Jeff W Lichtman,et al.  Why not connectomics? , 2013, Nature Methods.

[51]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[52]  B. Hochner.  An Embodied View of Octopus Neurobiology , 2012, Current Biology.

[53]  Kevin W Eliceiri,et al.  NIH Image to ImageJ: 25 years of image analysis , 2012, Nature Methods.

[54]  Johannes E. Schindelin,et al.  Fiji: an open-source platform for biological-image analysis , 2012, Nature Methods.

[55]  Daniel R. Berger,et al.  Pervasive Synaptic Branch Removal in the Mammalian Neuromuscular System at Birth , 2012, Neuron.

[56]  Kevin L. Briggman,et al.  Structural neurobiology: missing link to a mechanistic understanding of neural computation , 2012, Nature Reviews Neuroscience.

[57]  P. Strata,et al.  On the concept of spinocerebellum , 2012, Proceedings of the National Academy of Sciences of the United States of America.

[58]  W. Denk,et al.  The Big and the Small: Challenges of Imaging the Brain’s Circuits , 2011, Science.

[59]  Moritz Helmstaedter,et al.  High-accuracy neurite reconstruction for high-throughput neuroanatomy , 2011, Nature Neuroscience.

[60]  Arthur W. Wetzel,et al.  Network anatomy and in vivo physiology of visual cortical neurons , 2011, Nature.

[61]  B. Hochner,et al.  A new perspective on the organization of an invertebrate brain , 2011 .

[62]  Joseph F. Murray,et al.  Convolutional Networks Can Learn to Generate Affinity Graphs for Image Segmentation , 2010, Neural Computation.

[63]  J. Livet,et al.  A technicolour approach to the connectome , 2008, Nature Reviews Neuroscience.

[64]  B. Hochner,et al.  The Octopus Vertical Lobe Modulates Short-Term Learning Rate and Uses LTP to Acquire Long-Term Memory , 2008, Current Biology.

[65]  J C Fiala,et al.  Reconstruct: a free editor for serial section microscopy , 2005, Journal of microscopy.

[66]  S. Brenner,et al.  The structure of the nervous system of the nematode Caenorhabditis elegans. , 1986, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[67]  R. Cassada,et al.  The dauerlarva, a post-embryonic developmental variant of the nematode Caenorhabditis elegans. , 1975, Developmental biology.

[68]  M. Carandini,et al.  Suite 2 p : beyond 10 , 000 neurons with standard two-photon microscopy , 2016 .

[69]  L. Arranz,et al.  Network anatomy and in vivo physiology of mesenchymal stem and stromal cells , 2013 .

[70]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .