How Do Crowdworker Communities and Microtask Markets Influence Each Other? A Data-Driven Study on Amazon Mechanical Turk

Crowdworker online communities – operating in fora like mTurkForum and TurkerNation – are an important actor in microwork markets. Albeit central to market dynamics, how the behavior of crowdworker communities and the dynamics of online marketplaces influence each other is yet to be understood. To provide quantitative evidence of such influence, we performed an analysis on 6-years worth of mTurk market activities and community discussions in six fora. We investigated the nature of the relationships that exist between activities in fora, tasks published in mTurk, requesters for such tasks, and task completion speed. We validate – and expand upon – results from previous work by showing that (i) there are differences between market demand and community activities that are specific to fora and task types; (ii) the temporal progression of HIT availability in the market is predictive of the upcoming amount of crowdworker discussions, with significant differences across fora and discussion categories; (iii) activities in fora can have a significant positive impact on the completion speed of tasks available in the market.

[1]  Johan Bollen,et al.  Twitter mood predicts the stock market , 2010, J. Comput. Sci..

[2]  C. Granger.  Investigating causal relations by econometric models and cross-spectral methods , 1969 .

[3]  Gianluca Demartini,et al.  Pick-a-crowd: tell me what you like, and i'll tell you what to do , 2013, CIDR.

[4]  Panagiotis G. Ipeirotis,et al.  Demographics and Dynamics of Mechanical Turk Workers , 2018, WSDM.

[5]  Michael S. Bernstein,et al.  Daemo: A Self-Governed Crowdsourcing Marketplace , 2015, UIST.

[6]  Judith Redi,et al.  Modeling Task Complexity in Crowdsourcing , 2016, HCOMP.

[7]  Alessandro Bozzon,et al.  On the Improvement of Quality and Reliability of Trust Cues in Micro-task Crowdsourcing (Position paper) , 2017, ArXiv.

[8]  Ming Yin,et al.  The Communication Network Within the Crowd , 2016, WWW.

[9]  Yan Liu,et al.  Temporal causal modeling with graphical granger methods , 2007, KDD '07.

[10]  Jacki O'Neill,et al.  Being a turker , 2014, CSCW.

[11]  M. Six Silberman,et al.  Turkopticon: interrupting worker invisibility in amazon mechanical turk , 2013, CHI.

[12]  Mary L. Gray,et al.  The Crowd is a Collaborative Network , 2016, CSCW.

[13]  Dan Cosley,et al.  Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers' Experiences in Amazon Mechanical Turk , 2016, CHI.

[14]  Alessandro Bozzon,et al.  Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing , 2017, HT.

[15]  M. Eichler.  Causal inference in time series analysis , 2012 .

[16]  Stefan Dietze,et al.  A taxonomy of microtasks on the web , 2014, HT.

[17]  Yu Cui,et al.  A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters , 2017, CHI Extended Abstracts.

[18]  Alessandro Bozzon,et al.  Choosing the right crowd: expert finding in social networks , 2013, EDBT '13.

[19]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[20]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[21]  Gianluca Demartini,et al.  Scaling-Up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement , 2014, HCOMP.

[22]  Aditya G. Parameswaran,et al.  Finish Them!: Pricing Algorithms for Human Computation , 2014, Proc. VLDB Endow..

[23]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[24]  M. Six Silberman,et al.  Ethics and tactics of professional crowdwork , 2010, XRDS.

[25]  Bernhard Schölkopf,et al.  Causal Inference on Time Series using Restricted Structural Equation Models , 2013, NIPS.