Crowdsourcing Strategies for Text Creation Tasks

We examine deployment strategies for text translation and text summarization tasks. We formalize a deployment strategy along three dimensions: work structure, workforce organization , and work style. Work structure can be either simultaneous or sequential, workforce organization independent or collaborative, and work style either crowd-only or hybrid. We use Amazon Mechanical Turk to evaluate the cost, latency, and quality of various deployment strategies. We asses our strategies for different scenarios: short/long text, presence/absence of an outline, and popular/unpopular topics. Our findings serve as a basis to automate the deployment of text creation tasks.

[1]  Beng Chin Ooi,et al.  A hybrid machine-crowdsourcing system for matching web tables , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[2]  Adam Marcus,et al.  Argonaut: Macrotask Crowdsourcing for Complex Data Processing , 2015, Proc. VLDB Endow..

[3]  Gerard Salton,et al.  Automatic Text Structuring and Summarization , 1997, Inf. Process. Manag..

[4]  J. Mixter.  Fast , 2012 .

[5]  Aniket Kittur,et al.  Effects of simultaneous and sequential work structures on distributed collaborative interdependent tasks , 2014, CHI.

[6]  Hailong Sun,et al.  Combining Machine Learning and Crowdsourcing for Better Understanding Commodity Reviews , 2015, AAAI.

[7]  Guoliang Li,et al.  Crowdsourced Data Management: A Survey , 2016, IEEE Transactions on Knowledge and Data Engineering.

[8]  Jennifer Widom,et al.  Deco: declarative crowdsourcing , 2012, CIKM.

[9]  Tim Kraska,et al.  CrowdDB: answering queries with crowdsourcing , 2011, SIGMOD '11.

[10]  Andrés Monroy-Hernández,et al.  Storia: Summarizing Social Media Content based on Narrative Theory using Crowdsourcing , 2015, CSCW.

[11]  Rob Miller,et al.  Crowdsourced Databases: Query Processing with People , 2011, CIDR.

[12]  Benjamin Lecouteux,et al.  Towards accurate predictors of word quality for Machine Translation: Lessons learned on French-English and English-Spanish systems , 2015, Data Knowl. Eng..

[13]  Chris Callison-Burch,et al.  Crowdsourcing Translation: Professional Quality from Non-Professionals , 2011, ACL.

[14]  Chris Callison-Burch,et al.  Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk , 2009, EMNLP.