A game-based crowdsourcing platform for rapidly training middle and high school students to perform biomedical image analysis

We developed an easy-to-use and widely accessible crowd-sourcing tool for rapidly training humans to perform biomedical image diagnostic tasks and demonstrated this platform’s ability on middle and high school students in South Korea to diagnose malaria infected red-blood-cells (RBCs) using Giemsa-stained thin blood smears imaged under light microscopes. We previously used the same platform (i.e., BioGames) to crowd-source diagnostics of individual RBC images, marking them as malaria positive (infected), negative (uninfected), or questionable (insufficient information for a reliable diagnosis). Using a custom-developed statistical framework, we combined the diagnoses from both expert diagnosticians and the minimally trained human crowd to generate a gold standard library of malaria-infection labels for RBCs. Using this library of labels, we developed a web-based training and educational toolset that provides a quantified score for diagnosticians/users to compare their performance against their peers and view misdiagnosed cells. We have since demonstrated the ability of this platform to quickly train humans without prior training to reach high diagnostic accuracy as compared to expert diagnosticians. Our initial trial group of 55 middle and high school students has collectively played more than 170 hours, each demonstrating significant improvements after only 3 hours of training games, with diagnostic scores that match expert diagnosticians’. Next, through a national-scale educational outreach program in South Korea we recruited >1660 students who demonstrated a similar performance level after 5 hours of training. We plan to further demonstrate this tool’s effectiveness for other diagnostic tasks involving image labeling and aim to provide an easily-accessible and quickly adaptable framework for online training of new diagnosticians.

[1]  Aydogan Ozcan,et al.  Educational Games for Malaria Diagnosis , 2014, Science Translational Medicine.

[2]  E. Mcleod,et al.  Democratization of Nanoscale Imaging and Sensing Tools Using Photonics , 2015, Analytical chemistry.

[3]  Aydogan Ozcan,et al.  Mobile phones democratize and cultivate next-generation imaging, diagnostics and measurement tools. , 2014, Lab on a chip.

[4]  Aydogan Ozcan,et al.  Lensless Imaging and Sensing. , 2016, Annual review of biomedical engineering.

[5]  A. Ozcan,et al.  Synthetic aperture-based on-chip microscopy , 2015, Light: Science & Applications.

[6]  Zoran Popovic,et al.  Reconstructing the world in 3D: bringing games with a purpose outdoors , 2010, FDG.

[7]  Hongying Zhu,et al.  Optical imaging techniques for point-of-care diagnostics. , 2013, Lab on a chip.

[8]  Aydogan Ozcan,et al.  Field-Portable Pixel Super-Resolution Colour Microscope , 2013, PloS one.

[9]  Yun-En Liu,et al.  Gameplay analysis through state projection , 2010, FDG.

[10]  Hongying Zhu,et al.  Cost-effective and rapid blood analysis on a cell-phone. , 2013, Lab on a chip.

[11]  Aydogan Ozcan,et al.  Cellphone-based devices for bioanalytical sciences , 2014, Analytical and Bioanalytical Chemistry.

[12]  David Salesin,et al.  The challenge of designing scientific discovery games , 2010, FDG.

[13]  Aydogan Ozcan,et al.  High throughput on-chip analysis of high-energy charged particle tracks using lensfree imaging , 2015 .

[14]  Laura A. Dabbish,et al.  Designing games with a purpose , 2008, CACM.

[15]  Robert J Smith,et al.  Harnessing the crowd to accelerate molecular medicine research. , 2015, Trends in molecular medicine.

[16]  Aydogan Ozcan,et al.  Computational imaging, sensing and diagnostics for global health applications. , 2014, Current opinion in biotechnology.

[17]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[18]  A. Ozcan,et al.  Pixel super-resolution using wavelength scanning , 2015, Light: Science & Applications.

[19]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[20]  Yibo Zhang,et al.  Wide-field computational imaging of pathology slides using lens-free on-chip microscopy , 2014, Science Translational Medicine.

[21]  Steve Feng,et al.  Rapid imaging, detection and quantification of Giardia lamblia cysts using mobile-phone based fluorescent microscopy and machine learning. , 2015, Lab on a chip.

[22]  David Baker,et al.  Algorithm discovery by protein folding game players , 2011, Proceedings of the National Academy of Sciences.

[23]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[24]  Aydogan Ozcan,et al.  Increased space-bandwidth product in pixel super-resolved lensfree on-chip microscopy , 2013, Scientific Reports.

[25]  Aydogan Ozcan,et al.  Mobile Phone-Based Microscopy, Sensing, and Diagnostics , 2016, IEEE Journal of Selected Topics in Quantum Electronics.

[26]  A. Ozcan,et al.  Holographic pixel super-resolution in portable lensless on-chip microscopy using a fiber-optic array. , 2011, Lab on a chip.

[27]  Miguel Angel Luengo-Oroz,et al.  Crowdsourcing Malaria Parasite Quantification: An Online Game for Analyzing Images of Infected Thick Blood Smears , 2012, Journal of medical Internet research.

[28]  Sara Bennett,et al.  For Personal Use. Only Reproduce with Permission from Elsevier Ltd Overcoming Health-systems Constraints to Achieve the Millennium Development Goals , 2022 .

[29]  Steve Feng,et al.  Quantification of plant chlorophyll content using Google Glass. , 2015, Lab on a chip.

[30]  Aydogan Ozcan,et al.  Emerging Technologies for Next-Generation Point-of-Care Testing. , 2015, Trends in biotechnology.

[31]  Aydogan Ozcan,et al.  Toward giga-pixel nanoscopy on a chip: a computational wide-field look at the nano-scale without the use of lenses. , 2013, Lab on a chip.

[32]  Manuel Blum,et al.  reCAPTCHA: Human-Based Character Recognition via Web Security Measures , 2008, Science.

[33]  Zachary F. Meisel,et al.  Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review , 2013, Journal of General Internal Medicine.

[34]  Steve Feng,et al.  A Mathematical Framework for Combining Decisions of Multiple Experts toward Accurate and Remote Diagnosis of Malaria Using Tele-Microscopy , 2012, PloS one.

[35]  Bernardo A. Huberman.  Crowdsourcing and Attention , 2008, Computer.

[36]  Steve Feng,et al.  BioGames: A Platform for Crowd-Sourced Biomedical Image Analysis and Telediagnosis. , 2012, Games for health journal.

[37]  Derek K. Tseng,et al.  Imaging and sizing of single DNA molecules on a mobile phone. , 2014, ACS nano.

[38]  Steve Feng,et al.  Crowd-sourced BioGames: managing the big data problem for next-generation lab-on-a-chip platforms. , 2012, Lab on a chip.

[39]  Hongying Zhu,et al.  Calling Biomarkers in Milk Using a Protein Microarray on Your Smartphone , 2015, PloS one.

[40]  David Roman,et al.  Crowdsourcing and the question of expertise , 2009, Commun. ACM.

[41]  Aydogan Ozcan,et al.  High-throughput and label-free single nanoparticle sizing based on time-resolved on-chip microscopy. , 2015, ACS nano.

[42]  Steve Feng,et al.  Distributed Medical Image Analysis and Diagnosis through Crowd-Sourced Games: A Malaria Case Study , 2012, PloS one.

[43]  Yibo Zhang,et al.  Wide-field pathology imaging using on-chip microscopy , 2015, Virchows Archiv.

[44]  Z. Popovic,et al.  Crystal structure of a monomeric retroviral protease solved by protein folding game players , 2011, Nature Structural &Molecular Biology.

[45]  Steve Feng,et al.  Cellphone-Based Hand-Held Microplate Reader for Point-of-Care Testing of Enzyme-Linked Immunosorbent Assays. , 2015, ACS nano.

[46]  Benjamin M. Good,et al.  Crowdsourcing for bioinformatics , 2013, Bioinform..

[47]  Aydogan Ozcan,et al.  High-throughput analysis of horse sperms' 3D swimming patterns using computational on-chip imaging. , 2016, Animal reproduction science.