Aggregation Driven Progression System for GWAPs

As the uses of Games-With-A-Purpose (GWAPs) broadens, the systems that incorporate its usages have expanded in complexity. The types of annotations required within the NLP paradigm set such an example, where tasks can involve varying complexity of annotations. Assigning more complex tasks to more skilled players through a progression mechanism can achieve higher accuracy in the collected data while acting as a motivating factor that rewards the more skilled players. In this paper, we present the progression technique implemented in Wormingo , an NLP GWAP that currently includes two layers of task complexity. For the experiment, we have implemented four different progression scenarios on 192 players and compared the accuracy and engagement achieved with each scenario.

[1]  Juntao Yu,et al.  Progression in a Language Annotation Game with a Purpose , 2019, HCOMP.

[2]  Udo Kruschwitz,et al.  A Probabilistic Annotation Model for Crowdsourcing Coreference , 2018, EMNLP.

[3]  Udo Kruschwitz,et al.  Constructing an Anaphorically Annotated Corpus with Non-Experts: Assessing the Quality of Collaborative Annotations , 2009, PWNLP@IJCNLP.

[4]  Gert R. G. Lanckriet,et al.  User-centered design of a social game to tag music , 2009, HCOMP '09.

[5]  Bruno Guillaume,et al.  Creating Zombilingo, a game with a purpose for dependency syntax annotation , 2014, GamifIR '14.

[6]  Panagiotis G. Ipeirotis,et al.  Quizz: targeted crowdsourcing with a billion (potential) users , 2014, WWW.

[7]  William V. Wright,et al.  A Theory of Fun for Game Design , 2004 .

[8]  Christopher Cunningham,et al.  Gamification by Design - Implementing Game Mechanics in Web and Mobile Apps , 2011 .

[9]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[10]  Jon Chamberlain,et al.  Wormingo: a 'true gamification' approach to anaphoric annotation , 2019, FDG.

[11]  Udo Kruschwitz,et al.  Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation , 2013, TIIS.

[12]  Kuan-Ta Chen,et al.  The design of puzzle selection strategies for GWAP systems , 2010, Concurr. Comput. Pract. Exp..

[13]  Udo Kruschwitz,et al.  Phrase Detectives: A Web-based collaborative annotation game , 2008 .

[14]  D. Hoang.  FLOW: The Psychology of Optimal Experience , 2018 .