论文信息 - For a fistful of dollars: using crowd-sourcing to evaluate a spoken language CALL application

For a fistful of dollars: using crowd-sourcing to evaluate a spoken language CALL application

We present an evaluation of a Web-deployed spoken language CALL system, carried out using crowd-sourcing methods. The system, “Survival Japanese”, is a crash course in tourist Japanese implemented within the platform CALL-SLT. The evaluation was carried out over one week using the Amazon Mechanical Turk. Although we found a high proportion of attempted scammers, there was a core of 23 subjects who used the system in a responsible manner. The evidence that these subjects learned from their 111 sessions and 9092 spoken interactions was significant at P=0.001. Our conclusion is that crowd-sourcing is a potentially valid method for evaluating spoken CALL systems.

[1] Pierrette Bouillon,et al. Evaluating A Web-Based Spoken Language Translation Game For Learning Domain Language , 2011 .

[2] Manny Rayner,et al. A student-centred evaluation of a web-based spoken translation game , 2011, SLaTE.

[3] Hitoshi Isahara,et al. Many-to-Many Multilingual Medical Speech Translation on a PDA , 2008, AMTA 2008.

[4] Beth Ann Hockey,et al. Putting Linguistics into Speech Recognition: The Regulus Grammar Compiler (Studies in Computational Linguistics (Stanford, Calif.).) , 2006 .

[5] James R. Glass,et al. Collecting Voices from the Cloud , 2010, LREC.

[6] Cathy Chua,et al. What's the magic word? , 2010 .

[7] Johanna Gerlach,et al. A Multilingual CALL Game Based on Speech Translation , 2010, LREC.

[8] Chris Callison-Burch,et al. Creating Speech and Language Data With Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[9] Stephanie Seneff,et al. Automatic Assessment of Student Translations for Foreign Language Tutoring , 2007, NAACL.