This chapter evaluates the potential of the developed prototype to realize and study personalized task recommendation on existing crowdsourcing platforms and thereby improve the match between contributors and available tasks. To this intent, the following sections present a sequence of three studies that were conducted along the process of field testing and deploying the metacrowd service on the Mechanical Turk platform. Section 5.1 first describes a pilot study that gathered initial feedback from a small group of experienced contributors. Section 5.2 then lays out the details of a large-scale survey on general task search behavior and on the perceived utility of the metacrowd service. Section 5.3, finally, analyzes a dataset that was gathered during an extended period of productive use and discusses the challenges identified during the evaluation.
[1]
Lydia B. Chilton,et al.
Task search in a human computation market
,
2010,
HCOMP '10.
[2]
Guy Shani,et al.
Evaluating Recommendation Systems
,
2011,
Recommender Systems Handbook.
[3]
John Riedl,et al.
Explaining collaborative filtering recommendations
,
2000,
CSCW '00.
[4]
Martin Schader,et al.
Exploring task properties in crowdsourcing - an empirical study on mechanical turk
,
2011,
ECIS.
[5]
George Karypis,et al.
A Comprehensive Survey of Neighborhood-based Recommendation Methods
,
2011,
Recommender Systems Handbook.
[6]
C. Spearman.
The proof and measurement of association between two things. By C. Spearman, 1904.
,
1987,
The American journal of psychology.