Effect of Cognitive Abilities on Crowdsourcing Task Performance

Matching crowd workers to suitable tasks is highly desirable as it can enhance task performance, reduce the cost for requesters, and increase worker satisfaction. In this paper, we propose a method that considers workers’ cognitive ability to predict their suitability for a wide range of crowdsourcing tasks. We measure cognitive ability via fast-paced online cognitive tests with a combined average duration of 6.2 min. We then demonstrate that our proposed method can effectively assign or recommend workers to five different popular crowd tasks: Classification, Counting, Proofreading, Sentiment Analysis, and Transcription. Using our approach we demonstrate a significant improvement in the expected overall task accuracy. While previous methods require access to worker history or demographics, our work offers a quick and accurate way to determine which workers are more suitable for which tasks.

[1]  Kathryn M. McMillan,et al.  N‐back working memory paradigm: A meta‐analysis of normative functional neuroimaging studies , 2005, Human brain mapping.

[2]  A. Kristof-brown,et al.  CONSEQUENCES OF INDIVIDUALS' FIT AT WORK: A META-ANALYSIS OF PERSON-JOB, PERSON-ORGANIZATION, PERSON-GROUP, AND PERSON-SUPERVISOR FIT , 2005 .

[3]  Bipin Indurkhya,et al.  Cognitively inspired task design to improve user performance on crowdsourcing platforms , 2014, CHI.

[4]  Reynold Cheng,et al.  QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications , 2015, SIGMOD Conference.

[5]  George Karypis,et al.  Item-based top-N recommendation algorithms , 2004, TOIS.

[6]  Jorge Gonçalves,et al.  Motivating participation and improving quality of contribution in ubiquitous crowdsourcing , 2015, Comput. Networks.

[7]  Jeffrey R. Edwards,et al.  Person-job fit:: A conceptual integration, literature review, and methodological critique. , 1991 .

[8]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[9]  Gabriella Kazai,et al.  Quality Management in Crowdsourcing using Gold Judges Behavior , 2016, WSDM.

[10]  Alan C. Evans,et al.  Dissociation of human mid-dorsolateral from posterior dorsolateral frontal cortex in memory processing. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Jennifer Widom,et al.  Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace , 2017, Proc. VLDB Endow..

[12]  Gabriella Kazai,et al.  Worker types and personality traits in crowdsourcing relevance labels , 2011, CIKM '11.

[13]  Rolph E. Anderson,et al.  Multivariate Data Analysis (7th ed. , 2009 .

[14]  Terry A. Beehr,et al.  A meta-analysis of relations between person–organization fit and work attitudes , 2003 .

[15]  F. Schmidt,et al.  General mental ability in the world of work: occupational attainment and job performance. , 2004, Journal of personality and social psychology.

[16]  David Gross-Amblard,et al.  Using Hierarchical Skills for Optimized Task Assignment in Knowledge-Intensive Crowdsourcing , 2016, WWW.

[17]  Dan Cosley,et al.  Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers' Experiences in Amazon Mechanical Turk , 2016, CHI.

[18]  David J. Woehr,et al.  A QUANTITATIVE REVIEW OF THE RELATIONSHIP BETWEEN PERSON–ORGANIZATION FIT AND BEHAVIORAL OUTCOMES , 2006 .

[19]  Colin M. Macleod.  Half a century of research on the Stroop effect: an integrative review. , 1991, Psychological bulletin.

[20]  Albrecht Schmidt,et al.  Building Cognition-Aware Systems , 2017, UbiComp 2017.

[21]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[22]  Beng Chin Ooi,et al.  iCrowd: An Adaptive Crowdsourcing Framework , 2015, SIGMOD Conference.

[23]  Jessica B. Hamrick,et al.  psiTurk: An open-source framework for conducting replicable behavioral experiments online , 2016, Behavior research methods.

[24]  K. Nakayama,et al.  Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments , 2012, Psychonomic Bulletin & Review.

[25]  Joshua de Leeuw,et al.  jsPsych: A JavaScript library for creating behavioral experiments in a Web browser , 2014, Behavior Research Methods.

[26]  S. Gathercole,et al.  Executive functions and achievements in school: Shifting, updating, inhibition, and working memory , 2006, Quarterly journal of experimental psychology.

[27]  S. Monsell.  Task switching , 2003, Trends in Cognitive Sciences.

[28]  Jorge Gonçalves,et al.  Situated crowdsourcing using a market model , 2014, UIST.

[29]  Gabriella Kazai,et al.  The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy , 2012, CIKM.

[30]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[31]  Yannick Naudet,et al.  Personality Matters: Balancing for Personality Types Leads to Better Outcomes for Crowd Teams , 2016, CSCW.

[32]  John R Hodges,et al.  The Addenbrooke's Cognitive Examination Revised (ACE‐R): a brief cognitive test battery for dementia screening , 2006, International journal of geriatric psychiatry.

[33]  Jessica A. Grahn,et al.  Putting brain training to the test , 2010, Nature.

[34]  Amy L. Kristof.  PERSON-ORGANIZATION FIT: AN INTEGRATIVE REVIEW OF ITS CONCEPTUALIZATIONS, MEASUREMENT, AND IMPLICATIONS , 1996 .

[35]  P. Rabbitt,et al.  Cambridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. , 1994, Dementia.

[36]  Aniket Kittur,et al.  Instrumenting the crowd: using implicit behavioral measures to predict task performance , 2011, UIST.

[37]  Gabriella Kazai,et al.  In Search of Quality in Crowdsourcing for Search Engine Evaluation , 2011, ECIR.

[38]  Carsten Eickhoff,et al.  Cognitive Biases in Crowdsourcing , 2018, WSDM.

[39]  Shuguang Han,et al.  Crowdsourcing Human Annotation on Web Page Structure , 2016, ACM Trans. Intell. Syst. Technol..

[40]  P. Federico,et al.  Cognitive Styles, Abilities, and Aptitudes: Are They Dependent or Independent?. , 1984 .

[41]  Aniket Kittur,et al.  An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets , 2011, ICWSM.

[42]  Erika Borella,et al.  The Specific Role of Inhibition in Reading Comprehension in Good and Poor Comprehenders , 2010, Journal of learning disabilities.

[43]  Stefan Dietze,et al.  A taxonomy of microtasks on the web , 2014, HT.

[44]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[45]  Qiang Yang,et al.  Cross-task crowdsourcing , 2013, KDD.

[46]  Beng Chin Ooi,et al.  CDAS: A Crowdsourcing Data Analytics System , 2012, Proc. VLDB Endow..

[47]  Jorge Gonçalves,et al.  CrowdPickUp , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[48]  K. Miller.  Executive functions. , 2005, Pediatric annals.

[49]  C. Eriksen,et al.  Effects of noise letters upon the identification of a target letter in a nonsearch task , 1974 .

[50]  P. Scuffham,et al.  Patient Preferences and Willingness-To-Pay for a Home or Clinic Based Program of Chronic Heart Failure Management: Findings from the Which? Trial , 2013, PloS one.

[51]  Jorge Gonçalves,et al.  Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities , 2017, WWW.

[52]  Ruth B. Ekstrom,et al.  Manual for kit of factor-referenced cognitive tests , 1976 .

[53]  Jorge Gonçalves,et al.  Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours , 2013, UbiComp.

[54]  Fred J. Damerau,et al.  A technique for computer detection and correction of spelling errors , 1964, CACM.

[55]  Richard A. Cosier,et al.  Effects of cognitive styles and decision setting on performance , 1990 .

[56]  CHARLES E. Bailey.  Cognitive Accuracy and Intelligent Executive Function in the Brain and in Business , 2007, Annals of the New York Academy of Sciences.

[57]  K. Stanovich,et al.  Heuristics and Biases as Measures of Critical Thinking: Associations with Cognitive Ability and Thinking Dispositions , 2008 .

[58]  Deborah J. Armstrong,et al.  Person-Job Cognitive Style Fit for Software Developers: The Effect on Strain and Performance , 2005, J. Manag. Inf. Syst..

[59]  Gianluca Demartini,et al.  Pick-a-crowd: tell me what you like, and i'll tell you what to do , 2013, CIDR.