Crowdsourcing for Speech: Economic, Legal and Ethical analysis

With respect to spoken language resource production, Crowdsourcing - the process of distributing tasks to an open, unspecified population via the internet - offers a wide range of opportunities: populations with specific skills are potentially instantaneously accessible somewhere on the globe for any spoken language. As is the case for most newly introduced high-tech services, crowdsourcing raises both hopes and doubts, certainties and questions. A general analysis of Crowdsourcing for Speech processing could be found in (Eskenazi et al., 2013). This article will focus on ethical, legal and economic issues of crowdsourcing in general (Zittrain, 2008a) and of crowdsourcing services such as Amazon Mechanical Turk (Fort et al., 2011; Adda et al., 2011), a major platform for multilingual language resources (LR) production.

[1]  Panagiotis G. Ipeirotis,et al.  The Need for Standardization in Crowdsourcing , 2011 .

[2]  Vincent Berment,et al.  Méthodes pour informatiser les langues et les groupes de langues « peu dotées ». (Methods to computerize "little equipped" languages and groups of languages) , 2004 .

[3]  Ian McGraw,et al.  A self-labeling speech corpus: collecting spoken words with an online educational game , 2009, INTERSPEECH.

[4]  W. Buchholz.  The Ωμ+1-Rule , 1981 .

[5]  Alek Felstiner.  Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry , 2011 .

[6]  Praveen Paritosh,et al.  The anatomy of a large-scale human computation engine , 2010, HCOMP '10.

[7]  Bill Tomlinson,et al.  Who are the Turkers? Worker Demographics in Amazon Mechanical Turk , 2009 .

[8]  Norbert Reithinger,et al.  SpeechEval: A Domain-Independent User Simulation Platform for Spoken Dialog System Evaluation , 2011 .

[9]  Lydia B. Chilton,et al.  Task search in a human computation market , 2010, HCOMP '10.

[10]  Lydia B. Chilton,et al.  TurKit: human computation algorithms on mechanical turk , 2010, UIST.

[11]  Solomon Teferra Abate,et al.  An Amharic speech corpus for large vocabulary continuous speech recognition , 2005, INTERSPEECH.

[12]  Hermann Ney,et al.  Unsupervised training of acoustic models for large vocabulary continuous speech recognition , 2005, IEEE Transactions on Speech and Audio Processing.

[13]  Benjamin B. Bederson,et al.  Web workers unite! addressing challenges of online laborers , 2011, CHI Extended Abstracts.

[14]  Björn Hartmann,et al.  What's the Right Price? Pricing Tasks for Finishing on Time , 2011, Human Computation.

[15]  Laurent Besacier,et al.  Automatic Speech Recognition for Under-Resourced Languages: Application to Vietnamese Language , 2009, IEEE Transactions on Audio, Speech, and Language Processing.

[16]  Luis von Ahn.  Games with a Purpose , 2006, Computer.

[17]  Chris Callison-Burch,et al.  Creating Speech and Language Data With Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[18]  Christoph Draxler.  WWWTranscribe - a modular transcription system based on the world wide web , 1997, EUROSPEECH.

[19]  Ian McGraw,et al.  A self-transcribing speech corpus: collecting continuous speech with an online educational game , 2009, SLaTE.

[20]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[21]  M. Wendy Hennequin,et al.  The Future of the Internet and How to Stop It , 2011 .

[22]  Christoph Draxler,et al.  Phattsessionz: recording 1000 adolescent speakers in schools in Germany , 2005, INTERSPEECH.

[23]  Bengt Holmstrom,et al.  Moral Hazard and Observability , 1979 .

[24]  Awais Rashid,et al.  Software Engineering Ethics in a Digital World , 2009, Computer.

[25]  J. Zittrain,et al.  Ubiquitous human computing , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[26]  David L. Chen and William B. Dolan,et al.  Building a Persistent Workforce on Mechanical Turk for Multilingual Data Collection , 2011 .

[27]  A. Rustichini,et al.  Pay Enough or Don't Pay at All , 2000 .

[28]  Bill Tomlinson,et al.  Sellers' problems in human computation markets , 2010, HCOMP '10.

[29]  Chris Callison-Burch,et al.  Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription , 2010, NAACL.

[30]  Matthew Lease,et al.  Look before you leap: Legal pitfalls of crowdsourcing , 2011, ASIST.

[31]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[32]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[33]  Alexander I. Rudnicky,et al.  Using the Amazon Mechanical Turk for transcription of spoken language , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[34]  Panagiotis G. Ipeirotis.  Demographics of Mechanical Turk , 2010 .

[35]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[36]  Maxine Eskénazi,et al.  Toward better crowdsourced transcription: Transcription of a year of the Let's Go Bus Information System data , 2010, 2010 IEEE Spoken Language Technology Workshop.

[37]  Udo Kruschwitz,et al.  Phrase Detectives: A Web-based collaborative annotation game , 2008 .

[38]  K. Bretonnel Cohen,et al.  Last Words: Amazon Mechanical Turk: Gold Mine or Coal Mine? , 2011, CL.

[39]  Laurent Besacier,et al.  Developments of Swahili resources for an automatic speech recognition system , 2012, SLTU.

[40]  Benoît Sagot,et al.  Crowdsourcing for Language Resource Development: Critical Analysis of Amazon Mechanical Turk Overpowering Use , 2011, LTC 2011.

[41]  G. B. Varile.  Multilingual Speech Processing , 2005 .