A crowdsourcing semi-automatic image segmentation platform for cell biology

State-of-the-art computer-vision algorithms rely on big and accurately annotated data, which are expensive, laborious and time-consuming to generate. This task is even more challenging when it comes to microbiological images, because they require specialized expertise for accurate annotation. Previous studies show that crowdsourcing and assistive-annotation tools are two potential solutions to address this challenge. In this work, we have developed a web-based platform to enable crowdsourcing annotation of image data; the platform is powered by a semi-automated assistive tool to support non-expert annotators to improve the annotation efficiency. The behavior of annotators with and without the assistive tool is analyzed, using biological images of different complexity. More specifically, non-experts have been asked to use the platform to annotate microbiological images of gut parasites, which are compared with annotations by experts. A quantitative evaluation is carried out on the results, confirming that the assistive tools can noticeably decrease the non-expert annotation's cost (time, click, interaction, etc.) while preserving or even improving the annotation's quality. The annotation quality of non-experts has been investigated using IoU (intersection over union), precision and recall; based on this analysis we propose some ideas on how to better design similar crowdsourcing and assistive platforms.

[1]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[2]  Sanja Fidler,et al.  Annotating Object Instances with a Polygon-RNN , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Alison Q. O'Neil,et al.  Crowdsourcing Labels for Pathological Patterns in CT Lung Scans: Can Non-experts Contribute Expert-Quality Ground Truth? , 2017, CVII-STENT/LABELS@MICCAI.

[4]  Lena Maier-Hein,et al.  Clickstream Analysis for Crowd-Based Object Segmentation with Confidence , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  T. Peto,et al.  Crowdsourcing as a Novel Technique for Retinal Fundus Photography Classification: Analysis of Images in the EPIC Norfolk Cohort on Behalf of the UKBiobank Eye and Vision Consortium , 2013, PloS one.

[6]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Urs Ramer,et al.  An iterative procedure for the polygonal approximation of plane curves , 1972, Comput. Graph. Image Process..

[8]  Luc Van Gool,et al.  Deep Extreme Cut: From Extreme Points to Object Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Yugyung Lee,et al.  Utilizing Mask R-CNN for Detection and Segmentation of Oral Diseases , 2018, 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[10]  Nilanjan Ray,et al.  Training Convolutional Neural Networks and Compressed Sensing End-to-End for Microscopy Cell Detection , 2018, IEEE Transactions on Medical Imaging.

[11]  Gang Wang,et al.  Northeastern University , 2021, IEEE Pulse.

[12]  Jenny Chen,et al.  Opportunities for Crowdsourcing Research on Amazon Mechanical Turk , 2011 .

[13]  Ahmed Awad E. Ahmed,et al.  A New Biometric Technology Based on Mouse Dynamics , 2007, IEEE Transactions on Dependable and Secure Computing.

[14]  Heung-Il Suk,et al.  Deep Learning in Medical Image Analysis. , 2017, Annual review of biomedical engineering.

[15]  Sanja Fidler,et al.  Fast Interactive Object Annotation With Curve-GCN , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Sanja Fidler,et al.  Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++ , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[18]  Fei-Fei Li,et al.  What's the Point: Semantic Segmentation with Point Supervision , 2015, ECCV.

[19]  Raphael Sznitman,et al.  Expected Exponential Loss for Gaze-Based Video and Volume Ground Truth Annotation , 2017, CVII-STENT/LABELS@MICCAI.

[20]  James A. Hendler,et al.  Annotation and Provenance Tracking in Semantic Web Photo Libraries , 2006, IPAW.

[21]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Wonhee Lee,et al.  Multi-Task Self-Supervised Object Detection via Recycling of Bounding Box Annotations , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[24]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[25]  Paolo Napoletano,et al.  An interactive tool for manual, semi-automatic and automatic video annotation , 2015, Comput. Vis. Image Underst..

[26]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Chong Wang,et al.  Large-Scale Weakly Supervised Object Localization via Latent Category Learning , 2015, IEEE Transactions on Image Processing.

[28]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Hao Su,et al.  Crowdsourcing Annotations for Visual Object Detection , 2012, HCOMP@AAAI.

[30]  Deva Ramanan,et al.  Efficiently Scaling Up Video Annotation with Crowdsourced Marketplaces , 2010, ECCV.

[31]  Li Fei-Fei,et al.  Crowdsourcing in Computer Vision , 2016, Found. Trends Comput. Graph. Vis..

[32]  Frank Keller,et al.  Extreme Clicking for Efficient Object Annotation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Martin Jägersand,et al.  ByLabel: A Boundary Based Semi-Automatic Image Annotation Tool , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[34]  Lior Rokach,et al.  User identity verification via mouse dynamics , 2012, Inf. Sci..

[35]  Steve Feng,et al.  Distributed Medical Image Analysis and Diagnosis through Crowd-Sourced Games: A Malaria Case Study , 2012, PloS one.

[36]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[37]  Shadi Albarqouni,et al.  AggNet : Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images , 2016 .

[38]  Alberto Chávez-Aragón,et al.  A crowdsourcing web platform - hip joint segmentation by non-expert contributors , 2013, 2013 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[39]  Margrit Betke,et al.  Investigating the Influence of Data Familiarity to Improve the Design of a Crowdsourcing Image Annotation System , 2016, HCOMP.

[40]  Heikki Huttunen,et al.  Faster Bounding Box Annotation for Object Detection in Indoor Scenes , 2018, 2018 7th European Workshop on Visual Information Processing (EUVIP).

[41]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..