Digital video microscopy enhanced by deep learning

Single particle tracking is essential in many branches of science and technology, from the measurement of biomolecular forces to the study of colloidal crystals. Standard methods rely on algorithmic approaches; by fine-tuning several user-defined parameters, these methods can be highly successful at tracking a well-defined kind of particle under low-noise conditions with constant and homogenous illumination. Here, we introduce an alternative data-driven approach based on a convolutional neural network, which we name DeepTrack. We show that DeepTrack outperforms algorithmic approaches, especially in the presence of noise and under poor illumination conditions. We use DeepTrack to track an optically trapped particle under very noisy and unsteady illumination conditions, where standard algorithmic approaches fail. We then demonstrate how DeepTrack can also be used to track multiple particles and non-spherical objects such as bacteria, also at very low signal-to-noise ratios. In order to make DeepTrack readily available for other users, we provide a Python software package, which can be easily personalized and optimized for specific applications.

[1]  William J. Godinez,et al.  Objective comparison of particle tracking methods , 2014, Nature Methods.

[2]  Salman S Rogers,et al.  Precise particle tracking against a complicated background: polynomial fitting with Gaussian weight , 2007, Physical biology.

[3]  Jerry Chao,et al.  Quantitative study of single molecule location estimation techniques. , 2009, Optics express.

[4]  P. Koumoutsakos,et al.  Feature point tracking and trajectory analysis for video imaging in cell biology. , 2005, Journal of structural biology.

[5]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[7]  Nicolas Chenouard,et al.  Icy: an open bioimage informatics platform for extended reproducible research , 2012, Nature Methods.

[8]  R. Parthasarathy.  Rapid, accurate particle tracking by calculation of radial symmetry centers , 2012, Nature Methods.

[9]  D. Grier,et al.  Methods of Digital Video Microscopy for Colloidal Studies , 1996 .

[10]  Artyom Petrosyan,et al.  Energy flow between two hydrodynamically coupled particles kept at different effective temperatures , 2014, 1408.5319.

[11]  H. Berg,et al.  Migration of bacteria in semisolid agar. , 1989, Proceedings of the National Academy of Sciences of the United States of America.

[12]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[13]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[14]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[15]  Nishant Sule,et al.  Analysis and correction of errors in nanoscale particle tracking using the Single-pixel interior filling function (SPIFF) algorithm , 2017, Scientific Reports.

[16]  Suliana Manley,et al.  Single-particle tracking photoactivated localization microscopy for mapping single-molecule dynamics. , 2010, Methods in Enzymology.

[17]  Bo Li,et al.  Assembly and phase transitions of colloidal crystals , 2016, Nature Reviews Materials.

[18]  Mark D Hannel,et al.  Machine-learning techniques for fast and accurate feature localization in holograms of colloidal particles. , 2018, Optics express.

[19]  J. Adler,et al.  The effect of environmental conditions on the motility of Escherichia coli. , 1967, Journal of general microbiology.

[20]  Thomas Andrew Waigh,et al.  Advances in the microrheology of complex fluids , 2016, Reports on progress in physics. Physical Society.

[21]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[22]  S. Ram,et al.  Localization accuracy in single-molecule microscopy. , 2004, Biophysical journal.

[23]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[24]  François Chollet,et al.  Deep Learning with Python , 2017 .

[25]  Jörg Baumgartl,et al.  On the limits of digital video microscopy , 2005 .

[26]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[27]  J. Crocker,et al.  Multiple-particle tracking and two-point microrheology in cells. , 2007, Methods in cell biology.

[28]  John Tran,et al.  cuDNN: Efficient Primitives for Deep Learning , 2014, ArXiv.

[29]  Berin Martini,et al.  Large-Scale FPGA-based Convolutional Networks , 2011 .

[30]  Sean B Andersson,et al.  Localization of a fluorescent source without numerical fitting. , 2008, Optics express.

[31]  Quoc V. Le,et al.  Don't Decay the Learning Rate, Increase the Batch Size , 2017, ICLR.

[32]  Giovanni Volpe,et al.  Optical Tweezers: Principles and Applications , 2016 .

[33]  K. Neuman,et al.  Single-molecule force spectroscopy: optical tweezers, magnetic tweezers and atomic force microscopy , 2008, Nature Methods.

[34]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[35]  Sylvain Gigan,et al.  Disorder-mediated crowd control in an active matter system , 2016, Nature Communications.

[36]  M K Cheezum,et al.  Quantitative comparison of algorithms for tracking single fluorescent particles. , 2001, Biophysical journal.

[37]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  W. Steen.  Absorption and Scattering of Light by Small Particles , 1999 .

[39]  W. Webb,et al.  Precise nanometer localization analysis for individual fluorescent probes. , 2002, Biophysical journal.

[40]  S. Parveen,et al.  Nanoparticles: a boon to drug delivery, therapeutics, diagnostics and imaging. , 2012, Nanomedicine : nanotechnology, biology, and medicine.

[41]  M Gregory Forest,et al.  Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D , 2017, Proceedings of the National Academy of Sciences.