3D Structure From 2D Microscopy Images Using Deep Learning

Understanding the structure of a protein complex is crucial in determining its function. However, retrieving accurate 3D structures from microscopy images is highly challenging, particularly as many imaging modalities are two-dimensional. Recent advances in Artificial Intelligence have been applied to this problem, primarily using voxel based approaches to analyse sets of electron microscopy images. Here we present a deep learning solution for reconstructing the protein complexes from a number of 2D single molecule localization microscopy images, with the solution being completely unconstrained. Our convolutional neural network coupled with a differentiable renderer predicts pose and derives a single structure. After training, the network is discarded, with the output of this method being a structural model which fits the data-set. We demonstrate the performance of our system on two protein complexes: CEP152 (which comprises part of the proximal toroid of the centriole) and centrioles.

[1]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[2]  M. Nollmann,et al.  Angular reconstitution-based 3D reconstructions of nanomolecular structures from superresolution light-microscopy images , 2017, Proceedings of the National Academy of Sciences.

[3]  Charu C. Aggarwal,et al.  Neural Networks and Deep Learning , 2018, Springer International Publishing.

[4]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[5]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[6]  Matthias Zwicker,et al.  DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images , 2020, ICML.

[7]  Wan-Yen Lo,et al.  Accelerating 3D deep learning with PyTorch3D , 2019, SIGGRAPH Asia 2020 Courses.

[8]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Kyle M. Douglass,et al.  Homogeneous multifocal excitation for high-throughput super-resolution imaging , 2020, Nature Methods.

[10]  Mark Bates,et al.  Three dimensional particle averaging for structural imaging of macromolecular complexes by localization microscopy , 2019, bioRxiv.

[11]  Mauro Castelli,et al.  The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset , 2020, ICT Express.

[12]  Sriram Subramaniam,et al.  Cryo‐electron microscopy – a primer for the non‐microscopist , 2013, The FEBS journal.

[13]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[14]  S. Manley,et al.  High throughput 3D super-resolution microscopy reveals Caulobacter crescentus in vivo Z-ring organization , 2014, Proceedings of the National Academy of Sciences.

[15]  C. Aggarwal.  Neural Networks and Deep Learning: A Textbook , 2018 .

[16]  Alexey Dosovitskiy,et al.  Unsupervised Learning of Shape and Pose with Differentiable Point Clouds , 2018, NeurIPS.

[17]  Matthew D. Lew,et al.  Three-dimensional localization precision of the double-helix point spread function versus astigmatism and biplane. , 2010, Applied physics letters.

[18]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Suliana Manley,et al.  Homogeneous multifocal excitation for high-throughput super-resolution imaging , 2020, bioRxiv.

[20]  M. Zollhöfer,et al.  Pulsar: Efficient Sphere-based Neural Rendering , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Kyle M. Douglass,et al.  Multicolor single particle reconstruction of protein complexes , 2018, Nature Methods.

[22]  Vittorio Ferrari,et al.  Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading , 2019, International Journal of Computer Vision.

[23]  H. Leonhardt,et al.  A guide to super-resolution fluorescence microscopy , 2010, The Journal of cell biology.

[24]  David Baddeley,et al.  3D Multicolor Nanoscopy at 10,000 Cells a Day , 2019 .

[25]  Yun-Xing Wang,et al.  Molecular architecture of a cylindrical self-assembly at human centrosomes , 2019, Nature Communications.

[26]  Michael J. Black,et al.  OpenDR: An Approximate Differentiable Renderer , 2014, ECCV.