Co-destruction Patterns in Crowdsourcing

Crowdsourcing has been a successful paradigm in organising a large number of actors to work on specific tasks and contribute to knowledge collectively. However, the openness of such systems allows destructive patterns to form through actors’ dynamics. As a result, the collective effort of actors may not achieve the targeted objective due to lower engagement and lower quality contributions. There are varying forms of actor dynamics that can lead to suboptimal outcomes and this paper provides a systematic analysis of these in the form of a collection of patterns, derived from both the literature and from our own experiences with crowdsourcing systems. This collection of so-called co-destruction patterns allows for an-depth analysis of corwdsourcing systems which can benefit a comparative analysis and also assist with improvements of existing systems or the set-up of new ones. A survey reveals that these patterns have been observed in practice and are perceived as worthwhile addressing.

[1]  Jaime G. Carbonell,et al.  Detecting Non-Adversarial Collusion in Crowdsourcing , 2014, HCOMP.

[2]  Matthew Wilson,et al.  Crowdsourcing in a time of empowered stakeholders: Lessons from crowdsourcing campaigns , 2017 .

[3]  Xinxin Li,et al.  Salience Bias in Crowdsourcing Contests , 2017, Inf. Syst. Res..

[4]  Niki Pissinou,et al.  Approach to detect non-adversarial overlapping collusion in crowdsourcing , 2017, 2017 IEEE 36th International Performance Computing and Communications Conference (IPCCC).

[5]  F. Piller,et al.  Dark Side or Bright Light: Destructive and Constructive Deviant Content in Consumer Ideation Contests , 2017 .

[6]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[7]  Christoph Hube,et al.  Bias in Wikipedia , 2017, WWW.

[8]  Florian Lüdeke-Freund,et al.  Research on Sustainable Business Model Patterns: Status quo, Methodological Issues, and a Research Agenda , 2018, Sustainable Business Models.

[9]  Martin Wattenberg,et al.  Studying cooperation and conflict between authors with history flow visualizations , 2004, CHI.

[10]  Christian V. Baccarella,et al.  Social media? It's serious! Understanding the dark side of social media , 2018, European Management Journal.

[11]  Alessandro Lomi,et al.  The free encyclopedia that anyone can dispute: An analysis of the micro-structural dynamics of positive and negative relations in the production of contentious Wikipedia articles , 2020, Soc. Networks.

[12]  Carsten Eickhoff,et al.  Cognitive Biases in Crowdsourcing , 2018, WSDM.

[13]  Loïc Plé,et al.  Why Do We Need Research on Value Co-destruction? , 2017 .

[14]  James D. Herbsleb,et al.  Influence of social and technical factors for evaluating contribution in GitHub , 2014, ICSE.

[15]  Abbe Mowshowitz,et al.  Bias on the web , 2002, CACM.

[16]  A. Smith,et al.  The value co-destruction process: a customer resource perspective , 2013 .

[17]  Gabriella Kazai,et al.  Worker types and personality traits in crowdsourcing relevance labels , 2011, CIKM '11.

[18]  Mehmet A. Orgun,et al.  A Proof-of-Trust Consensus Protocol for Enhancing Accountability in Crowdsourcing Services , 2019, IEEE Transactions on Services Computing.

[19]  Ulrik Brandes,et al.  Visual Analysis of Controversy in User-Generated Encyclopedias∗ , 2007, 2007 IEEE Symposium on Visual Analytics Science and Technology.

[20]  Efraim Turban,et al.  What can crowdsourcing do for decision support? , 2014, Decis. Support Syst..

[21]  Max Jacobson,et al.  A Pattern Language: Towns, Buildings, Construction , 1981 .

[22]  Rajiv Kishore,et al.  Rules of Crowdsourcing: Models, Issues, and Systems of Control , 2013, Inf. Syst. Manag..

[23]  Pascal Neis,et al.  Towards Automatic Vandalism Detection in OpenStreetMap , 2012, ISPRS Int. J. Geo Inf..

[24]  Stefan Dietze,et al.  Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks , 2017, ACM Trans. Comput. Hum. Interact..

[25]  Guevara Noubir,et al.  On GitHub's Programming Languages , 2016, ArXiv.

[26]  Martin Fowler,et al.  Analysis patterns - reusable object models , 1996, Addison-Wesley series in object-oriented software engineering.

[27]  Qiang Liu,et al.  Scoring Workers in Crowdsourcing: How Many Control Questions are Enough? , 2013, NIPS.

[28]  Yejin Choi,et al.  Language of Vandalism: Improving Wikipedia Vandalism Detection via Stylometric Analysis , 2011, ACL.

[29]  Paolo Rosso,et al.  Wikipedia Vandalism Detection: Combining Natural Language, Metadata, and Reputation Features , 2011, CICLing.

[30]  Arvind Malhotra,et al.  The dark side of the sharing economy … and how to lighten it , 2014, Commun. ACM.

[31]  Imad H. Elhajj,et al.  Modelling Cognitive Bias in Crowdsourcing Systems , 2019, Cognitive Systems Research.

[32]  Eric Horvitz,et al.  Identifying and Accounting for Task-Dependent Bias in Crowdsourcing , 2015, HCOMP.

[33]  Maja Bott,et al.  The Role of Crowdsourcing for Better Governance in International Development , 2012 .

[34]  HyunYong Lee,et al.  Simplified clique detection for collusion-resistant reputation management scheme in P2P networks , 2010, 2010 10th International Symposium on Communications and Information Technologies.

[35]  Jan Marco Leimeister,et al.  Rags to Riches - How Signaling Behaviour Causes a Power Shift in Crowdsourcing Markets , 2016, ECIS.

[36]  Kyumin Lee,et al.  Characterizing and automatically detecting crowdturfing in Fiverr and Twitter , 2015, Social Network Analysis and Mining.

[37]  Martin Potthast,et al.  Crowdsourcing a wikipedia vandalism corpus , 2010, SIGIR.

[38]  Elisa Bertino,et al.  Collusion Detection in Online Rating Systems , 2013, APWeb.

[39]  Renato Lo Cigno,et al.  Collusion in peer-to-peer systems , 2011, Comput. Networks.

[40]  Ruben Chumpitaz Cáceres,et al.  Not always co-creation: introducing interactional co-destruction of value in service-dominant logic , 2010 .

[41]  Santiago Moisés Mola-Velasco,et al.  Wikipedia vandalism detection , 2011, WWW.

[42]  Panagiotis G. Ipeirotis,et al.  The Global Opportunity in Online Outsourcing , 2015 .

[43]  Eileen M. Trauth,et al.  Hanging with the right crowd: Matching crowdsourcing need to crowd characteristics , 2012, AMCIS.

[44]  Premkumar T. Devanbu,et al.  Wait for It: Determinants of Pull Request Evaluation Latency on GitHub , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[45]  András Kornai,et al.  Edit Wars in Wikipedia , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[46]  Michael I. Jordan,et al.  Bayesian Bias Mitigation for Crowdsourcing , 2011, NIPS.

[47]  Nicolas Hidalgo,et al.  Crowdsourcing Under Attack: Detecting Malicious Behaviors in Waze , 2018, IFIPTM.

[48]  Henri Simula,et al.  The Rise and Fall of Crowdsourcing? , 2013, 2013 46th Hawaii International Conference on System Sciences.

[49]  Boualem Benatallah,et al.  Quality Control in Crowdsourcing , 2018, ACM Comput. Surv..

[50]  Hailong Sun,et al.  Collusion-Proof Result Inference in Crowdsourcing , 2018, Journal of Computer Science and Technology.

[51]  Jure Leskovec,et al.  The dynamics of viral marketing , 2005, EC '06.

[52]  Kyumin Lee,et al.  The Dark Side of Micro-Task Marketplaces: Characterizing Fiverr and Automatically Detecting Crowdturfing , 2014, ICWSM.

[53]  Ayushi Rastogi,et al.  Do Biases Related to Geographical Location Influence Work-Related Decisions in GitHub? , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[54]  Alessandro Lomi,et al.  The Third Man: hierarchy formation in Wikipedia , 2017, Applied Network Science.

[55]  Tim Hughes,et al.  Antecedents to value diminution , 2016 .