Quantius: Generic, high-fidelity human annotation of scientific images at 105-clicks-per-hour

We describe Quantius, a crowd-based image annotation platform that provides an accurate alternative to task-specific computational algorithms for difficult image analysis problems. We use Quantius to quantify a variety of computationally challenging medium-throughput tasks with ~50x and 30x savings in analysis time and cost respectively, relative to a single expert annotator. We show equivalent deep learning performance for Quantius- and expert-derived annotations, bridging towards scalable integration with tailored machine-learning algorithms.

[1]  Zachary F. Meisel,et al.  Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review , 2013, Journal of General Internal Medicine.

[2]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[3]  H. Sauermann,et al.  Crowd science user contribution patterns and their implications , 2015, Proceedings of the National Academy of Sciences.

[4]  Frank Jülicher,et al.  Multimotor transport in a system of active and inactive kinesin-1 motors. , 2014, Biophysical journal.

[5]  Shumin Zhai,et al.  Speed-accuracy tradeoff in Fitts' law tasks-on the equivalency of actual and nominal pointing precision , 2004, Int. J. Hum. Comput. Stud..

[6]  David De Roure,et al.  Zooniverse: observing the world's largest citizen science platform , 2014, WWW.

[7]  Michael P. Sheetz,et al.  Identification of a novel force-generating protein, kinesin, involved in microtubule-based motility , 1985, Cell.

[8]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[9]  In-Kwon Lee,et al.  Curve reconstruction from unorganized points , 2000, Comput. Aided Geom. Des..

[10]  R. J. Adcock.  A Problem in Least Squares , 1878 .

[11]  David Zwicker,et al.  Tracking single particles and elongated filaments with nanometer precision. , 2011, Biophysical journal.

[12]  Andrew Zisserman,et al.  Detecting overlapping instances in microscopy images using extremal region trees , 2016, Medical Image Anal..

[13]  C. Gilbert,et al.  Learning to find a shape , 2000, Nature Neuroscience.

[14]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[15]  David N Breslauer,et al.  Mobile Phone Based Clinical Microscopy for Global Health Applications , 2009, PloS one.

[16]  Jason P. Gleghorn,et al.  Mechanically patterning the embryonic airway epithelium , 2015, Proceedings of the National Academy of Sciences.

[17]  Stephen L. Chiu,et al.  Fuzzy Model Identification Based on Cluster Estimation , 1994, J. Intell. Fuzzy Syst..

[18]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[19]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[20]  O. Kiehn,et al.  Dual-mode operation of neuronal networks involved in left–right alternation , 2013, Nature.

[21]  Manuel Corpas,et al.  Lessons from Fraxinus, a crowd-sourced citizen science game in genomics , 2015, eLife.

[22]  Edward S Boyden,et al.  Nanoscale Imaging of RNA with Expansion Microscopy , 2016, Nature Methods.

[23]  Andrew Zisserman,et al.  Microscopy cell counting and detection with fully convolutional regression networks , 2018, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[24]  Soe-Tsyr Daphne Yuan,et al.  An impactful crowdsourcing intermediary design - a case of a service imagery crowdsourcing system , 2018, Inf. Syst. Frontiers.

[25]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[26]  S. Chiu,et al.  A cluster estimation method with extension to fuzzy model identification , 1994, Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference.

[27]  I. Sbalzarini,et al.  High‐resolution cell outline segmentation and tracking from phase‐contrast microscopy images , 2012, Journal of microscopy.

[28]  Pietro Michelucci,et al.  The power of crowds , 2016, Science.

[29]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[30]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[31]  Christopher J. Obara,et al.  Increased spatiotemporal resolution reveals highly dynamic dense tubular matrices in the peripheral ER , 2016, Science.

[32]  Andrea Vedaldi,et al.  Vlfeat: an open and portable library of computer vision algorithms , 2010, ACM Multimedia.

[33]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[34]  Lydia B. Chilton,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[35]  Srinivas C. Turaga,et al.  Space-time wiring specificity supports direction selectivity in the retina , 2014, Nature.

[36]  Johannes Schindelin,et al.  TrackMate: An open and extensible platform for single-particle tracking. , 2017, Methods.

[37]  F. Markowetz,et al.  Quantitative Image Analysis of Cellular Heterogeneity in Breast Tumors Complements Genomic Profiling , 2012, Science Translational Medicine.

[38]  Minjung Kang,et al.  Stem Cell Reports , Volume 2 Supplemental Information A Rapid and Efficient 2 D / 3 D Nuclear Segmentation Method for Analysis of Early Mouse Embryo and Stem Cell Image Data , 2014 .

[39]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[40]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.