Weakly-Supervised Cell Tracking via Backward-and-Forward Propagation

We propose a weakly-supervised cell tracking method that can train a convolutional neural network (CNN) by using only the annotation of "cell detection" (i.e., the coordinates of cell positions) without association information, in which cell positions can be easily obtained by nuclear staining. First, we train co-detection CNN that detects cells in successive frames by using weak-labels. Our key assumption is that co-detection CNN implicitly learns association in addition to detection. To obtain the association, we propose a backward-and-forward propagation method that analyzes the correspondence of cell positions in the outputs of co-detection CNN. Experiments demonstrated that the proposed method can associate cells by analyzing co-detection CNN. Even though the method uses only weak supervision, the performance of our method was almost the same as the state-of-the-art supervised method. Code is publicly available in this https URL

[1]  Petr Matula,et al.  DIC Image Segmentation of Dense Cell Populations by Combining Deep Learning and Watershed , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[2]  Suha Kwak,et al.  Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Xiaoxu Wang,et al.  CELL SEGMENTATION AND TRACKING USING TEXTURE-ADAPTIVE SNAKES , 2007, 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[4]  Sergio Guadarrama,et al.  Tracking Emerges by Colorizing Videos , 2018, ECCV.

[5]  Wiro J. Niessen,et al.  Bayesian tracking for fluorescence microscopic imaging , 2006, 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 2006..

[6]  Ryoma Bise,et al.  Cell Tracking with Deep Learning for Cell Detection and Motion Estimation in Low-Frame-Rate , 2019, MICCAI.

[7]  Huchuan Lu,et al.  GradNet: Gradient-Guided Network for Visual Object Tracking , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  James J. Little,et al.  A Boosted Particle Filter: Multitarget Detection and Tracking , 2004, ECCV.

[9]  Philip H. S. Torr,et al.  Weakly- and Semi-Supervised Panoptic Segmentation , 2022 .

[10]  Silvio Savarese,et al.  Object Co-detection , 2012, ECCV.

[11]  Bernt Schiele,et al.  Simple Does It: Weakly Supervised Instance and Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Xinlei Chen,et al.  PixelNet: Representation of the pixels, by the pixels, and for the pixels , 2017, ArXiv.

[13]  Didier Mutter,et al.  Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos , 2018, International Journal of Computer Assisted Radiology and Surgery.

[14]  Mary Brady,et al.  Lineage mapper: A versatile cell and particle tracker , 2016, Scientific Reports.

[15]  Allan Jabri,et al.  Learning Correspondence From the Cycle-Consistency of Time , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Olaf Ronneberger,et al.  Cell segmentation and tracking in phase contrast images using graph cut with asymmetric boundary costs , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[17]  Shanshan Tu,et al.  Multiple instance deep learning for weakly-supervised visual object tracking , 2020, Signal Process. Image Commun..

[18]  Thomas Neff,et al.  Instance Segmentation and Tracking with Cosine Embeddings and Recurrent Hourglass Networks , 2018, MICCAI.

[19]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[20]  Suha Kwak,et al.  Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Wei Liu,et al.  Unsupervised Deep Tracking , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[23]  Lee E. Weiss,et al.  Phase contrast time-lapse microscopy datasets with automated and manual cell tracking annotations , 2018, Scientific Data.

[24]  Fred A. Hamprecht,et al.  Conservation Tracking , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Ryoma Bise,et al.  MPM: Joint Representation of Motion and Position Map for Cell Tracking , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Vineeth N. Balasubramanian,et al.  Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[27]  Martin Wattenberg,et al.  SmoothGrad: removing noise by adding noise , 2017, ArXiv.

[28]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Takeo Kanade,et al.  Cell population tracking and lineage construction with spatiotemporal context , 2008, Medical Image Anal..

[30]  Takeo Kanade,et al.  Understanding the phase contrast optics to restore artifact-free microscopy images for segmentation , 2012, Medical Image Anal..

[31]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[32]  Takeo Kanade,et al.  Cell image analysis: Algorithms, system and applications , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[33]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[34]  Klaus-Robert Müller,et al.  Learning how to explain neural networks: PatternNet and PatternAttribution , 2017, ICLR.

[35]  Takeo Kanade,et al.  Cell segmentation in phase contrast microscopy images via semi-supervised classification over optics-related features , 2013, Medical Image Anal..

[36]  Ryoma Bise,et al.  Weakly Supervised Cell Instance Segmentation by Propagating from Detection Response , 2019, MICCAI.

[37]  Peng Gao,et al.  Joint Multi-frame Detection and Segmentation for Multi-cell Tracking , 2019, ICIG.

[38]  Takeo Kanade,et al.  Reliably Tracking Partially Overlapping Neural Stem Cells in DIC Microscopy Image Sequences , 2009 .

[39]  Takeo Kanade,et al.  Reliable cell tracking by global data association , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[40]  David Barber,et al.  Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Zhe L. Lin,et al.  Top-Down Neural Attention by Excitation Backprop , 2016, International Journal of Computer Vision.

[42]  Alexander Binder,et al.  Explaining nonlinear classification decisions with deep Taylor decomposition , 2015, Pattern Recognit..

[43]  Wen Gao,et al.  Visual tracking via weakly supervised learning from multiple imperfect oracles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[44]  Bjoern H. Menze,et al.  Cell Lineage Tracing in Lens-Free Microscopy Videos , 2017, MICCAI.

[45]  Juho Kannala,et al.  Joint cell segmentation and tracking using cell proposals , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[46]  Masahiro Kino-oka,et al.  Cell tracking under high confluency conditions by candidate cell region detection-based association approach , 2013 .

[47]  Bjoern H. Menze,et al.  Tracing cell lineages in videos of lens‐free microscopy , 2018, Medical Image Anal..

[48]  Nathalie Harder,et al.  An Objective Comparison of Cell Tracking Algorithms , 2017, Nature Methods.

[49]  Nathalie Harder,et al.  A benchmark for comparison of cell tracking algorithms , 2014, Bioinform..

[50]  Milan Sonka,et al.  Cell Segmentation, Tracking, and Mitosis Detection Using Temporal Context , 2005, MICCAI.

[51]  Alexander Binder,et al.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation , 2015, PloS one.