Effects of Expertise Assessment on the Quality of Task Routing in Human Computation

Human computation systems are characterized by the use of human workers to solve computationally difficult problems. Expertise profiling involves assessment and representation of a worker’s expertise, in order to route human computation tasks to appropriate workers. This paper studies the relationship between the assessment workload on workers and the quality of task routing. Three expertise assessment approaches were compared with the help of a user study, using two different groups of human workers. The first approach requests workers to provide selfassessment of their knowledge. The second approach measures the knowledge of workers through their performance against tasks with known responses. We propose a third approach based on a combination of selfassessment and task-assessment. The results suggest that the self-assessment approach requires minimum assessment workload from workers during expertise profiling. By comparison, the task-assessment approach achieved the highest response rate and accuracy. The proposed approach requires less assessment workload, while achieving the response rate and accuracy similar to the task-assessment approach. Author Keywords Human computation, expertise assessment, task routing

[1]  Tom Heath,et al.  Linked Data: Evolving the Web into a Global Data Space , 2011, Linked Data.

[2]  David J. Weiss,et al.  Empirical Assessment of Expertise , 2003, Hum. Factors.

[3]  Luis von Ahn Human Computation , 2008, ICDE.

[4]  John Riedl,et al.  SuggestBot: using intelligent task routing to help people find work in wikipedia , 2007, IUI '07.

[5]  Paul N. Bennett,et al.  The effects of choice in routing relevance judgments , 2011, SIGIR '11.

[6]  Krzysztof Z. Gajos,et al.  Human computation tasks with global constraints , 2012, CHI.

[7]  Michael D. Lee,et al.  Inferring Expertise in Knowledge and Prediction Ranking Tasks , 2012, Top. Cogn. Sci..

[8]  David J. Weiss,et al.  Performance-based assessment of expertise: How to decide if someone is an expert or not , 2002, Eur. J. Oper. Res..

[9]  M. de Rijke,et al.  Expertise Retrieval , 2012, Found. Trends Inf. Retr..

[10]  Eric Horvitz,et al.  Task routing for prediction tasks , 2012, AAMAS.

[11]  Junjie Yao,et al.  Routing Questions to the Right Users in Online Communities , 2009, 2009 IEEE 25th International Conference on Data Engineering.

[12]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[13]  Edward Curry,et al.  Towards Expertise Modelling for Routing Data Cleaning Tasks within a Community of Knowledge Workers , 2012, ICIQ.

[14]  Rob Miller,et al.  Crowdsourced Databases: Query Processing with People , 2011, CIDR.

[15]  Christian Bizer,et al.  Evolving the Web into a Global Data Space , 2011, BNCOD.

[16]  Edward Curry,et al.  Leveraging matching dependencies for guided user feedback in linked data applications , 2012, IIWeb '12.