Crowdsourcing and Massively Collaborative Science: A Systematic Literature Review and Mapping Study

Current times are denoting unprecedented indicators of scientific data production, and the involvement of the wider public (the crowd) on research has attracted increasing attention. Drawing on review of extant literature, this paper outlines some ways in which crowdsourcing and mass collaboration can leverage the design of intelligent systems to keep pace with the rapid transformation of scientific work. A systematic literature review was performed following the guidelines of evidence-based software engineering and a total of 148 papers were identified as primary after querying digital libraries. From our review, a lack of methodological frameworks and algorithms for enhancing interactive intelligent systems by combining machine and crowd intelligence is clearly manifested and we will need more technical support in the future. We lay out a vision for a cyberinfrastructure that comprises crowd behavior, task features, platform facilities, and integration of human inputs into AI systems.

[1]  Krzysztof Z. Gajos,et al.  Crowdsourcing as a Tool for Research: Implications of Uncertainty , 2017, CSCW.

[2]  Clarence A. Ellis,et al.  Groupware: some issues and experiences , 1991, CACM.

[3]  Jano Moreira de Souza,et al.  CSCWD: Five characters in search of crowds , 2012, Proceedings of the 2012 IEEE 16th International Conference on Computer Supported Cooperative Work in Design (CSCWD).

[4]  Benjamin M. Good,et al.  Microtask Crowdsourcing for Disease Mention Annotation in PubMed Abstracts , 2014, Pacific Symposium on Biocomputing.

[5]  Qinghua Zhu,et al.  Evaluation on crowdsourcing research: Current status and future direction , 2012, Information Systems Frontiers.

[6]  Benjamin M. Good,et al.  Games with a scientific purpose , 2011, Genome Biology.

[7]  Wai-Tat Fu,et al.  Don't hide in the crowd!: increasing social transparency between peer workers improves crowdsourcing outcomes , 2013, CHI.

[8]  Ece Kamar,et al.  Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence , 2016, IJCAI.

[9]  D. Allison,et al.  Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example , 2014, PloS one.

[10]  Alexander Klippel,et al.  PITFALLS AND POTENTIALS OF CROWD SCIENCE: A META-ANALYSIS OF CONTEXTUAL INFLUENCES , 2015 .

[11]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[12]  Marina Jirotka,et al.  Supporting Scientific Collaboration: Methods, Tools and Concepts , 2013, Computer Supported Cooperative Work (CSCW).

[13]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[14]  Gerhard Weikum,et al.  Combining information extraction and human computing for crowdsourced knowledge acquisition , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[15]  J. Hendler,et al.  Amplify scientific discovery with artificial intelligence , 2014, Science.

[16]  Zachary F. Meisel,et al.  Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review , 2013, Journal of General Internal Medicine.

[17]  Robert Johansen,et al.  Groupware: Computer Support for Business Teams , 1988 .

[18]  Lydia B. Chilton,et al.  Community Clustering: Leveraging an Academic Crowd to Form Coherent Conference Sessions , 2013, HCOMP.

[19]  Lesandro Ponciano,et al.  Volunteers' Engagement in Human Computation for Astronomy Projects , 2014, Computing in Science & Engineering.

[20]  Rolf Pfeifer,et al.  Crowdsourcing, Open Innovation and Collective Intelligence in the Scientific Method - A Research Agenda and Operational Framework , 2010, ALIFE.

[21]  Max J. Pfeffer,et al.  Process, Not Product: Investigating Recommendations for Improving Citizen Science “Success” , 2013, PloS one.

[22]  E. Deci,et al.  The general causality orientations scale: Self-determination in personality , 1985 .

[23]  Pirjo Friedrich,et al.  Massively distributed authorship of academic papers , 2012, CHI Extended Abstracts.

[24]  Jeffrey V. Nickerson,et al.  The Crowdsourcing Design Space , 2011, HCI.

[25]  Daniela Cruzes,et al.  Research synthesis in software engineering: A tertiary study , 2011, Inf. Softw. Technol..

[26]  David P. Anderson,et al.  Scientists@Home: What Drives the Quantity and Quality of Online Citizen Science Participation? , 2014, PloS one.

[27]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[28]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[29]  Rob Miller,et al.  Real-time crowd control of existing interfaces , 2011, UIST.

[30]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[31]  Jano Moreira de Souza,et al.  Mix4Crowds - Toward a framework to design crowd collaboration with science , 2013, Proceedings of the 2013 IEEE 17th International Conference on Computer Supported Cooperative Work in Design (CSCWD).

[32]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[33]  Jesse Chandler,et al.  Risks and Rewards of Crowdsourcing Marketplaces , 2014, Handbook of Human Computation.

[34]  Aniket Kittur,et al.  Apolo: making sense of large network data by combining rich user interaction and machine learning , 2011, CHI.

[35]  Alimohammad Shahri,et al.  Crowdsourcing: A taxonomy and systematic mapping study , 2015, Comput. Sci. Rev..

[36]  Matthew Lease,et al.  On Quality Control and Machine Learning in Crowdsourcing , 2011, Human Computation.

[37]  Tok Wang Ling,et al.  Using hybrid algorithmic-crowdsourcing methods for academic knowledge acquisition , 2017, Cluster Computing.

[38]  Stefan Dietze,et al.  Human Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing , 2015, IEEE Intelligent Systems.