Crowdsourcing for linguistic field research and e-learning

Crowdsourcing denotes the transfer of work commonly carried out by single humans to a large group of people. Nowadays, crowdsourcing is employed for many purposes, like people contributing their knowledge to Wikipedia, researchers predicting diseases from data on Twitter, or players solving protein folding problems in games. Still, there are areas for which the application of crowdsourcing has not yet been investigated thoroughly. This thesis examines crowdsourcing for two such areas: for empirical research in sciences oriented on humans -focusing on linguistic field research- and for e-learning. Sciences oriented on humans -like linguistics, sociology, or art history- depend on empirical research. For example, in traditional linguistic field research researchers ask questions and fill in forms. Such methods are time-consuming, costly, and not free of biases. This thesis proposes the application of crowdsourcing techniques to overcome these disadvantages and to support empirical research in getting more efficient. Therefore, the concept of a generic market for trading with symbolic goods and speculating on their characteristics in a playful manner, called Agora is introduced. Agora aims to be an "operating system" for social media applications gathering data. Furthermore, the Web-based crowdsourcing platform metropolitalia has been established for hosting two social media applications based upon Agora: Mercato Linguistico and Poker Parole. These applications have been conceived as part of this thesis for gathering complementary data and meta-data on Italian language varieties. Mercato Linguistico incites players to express their own knowledge or beliefs, Poker Parole incites players to make conjectures on the contributions of others. Thereby the primary meta-data collected with Mercato Linguistico are enriched with secondary, reflexive meta-data from Poker Parole, which are needed for studies on the perception of languages. An evaluation of the data gathered on metropolitalia exhibits the viability of the market-based approach of Agora and highlights its strengths. E-learning is concerned with the use of digital technology for learning, nowadays especially via the Internet. This thesis investigates how e-learning applications can support students with association-based learning and lecturers with teaching. For that, a game-like e-learning tool named Termina is proposed in this thesis. From the data collected with Termina association maps are constructed. An association map is a simplified version of a concept map, in which concepts are represented as rectangles and relationships between concepts as links. They constitute an abstract comprehension of a topic. Students profit from the association maps' availability, learn from other participating students, and can track their own learning progress. Lecturers gain insights into the knowledge and into potential misunderstandings of their students. An evaluation of Termina and the collected data along a university course exhibits Termina's usefulness for both students and lecturers. The main contributions of this thesis are (1) a literature review over collective intelligence, crowdsourcing, and related fields, (2) a model of a generic market for gathering data for empirical research efficiently, (3) two applications based on this model and results of an evaluation of the data gathered with them, (4) the game-like e-learning tool Termina together with insights from its evaluation, and (5) a generic software architecture for all aforementioned applications.

[1]  George Siemens,et al.  Penetrating the fog: analytics in learning and education , 2014 .

[2]  Luis von Ahn,et al.  Human computation , 2009, 2009 46th ACM/IEEE Design Automation Conference.

[3]  Lydia B. Chilton,et al.  TurKit: Tools for iterative tasks on mechanical turk , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[4]  J. Keynes,et al.  The General Theory of Employment, Interest and Money. , 1936 .

[5]  David M. Pennock,et al.  Prediction Markets: Does Money Matter? , 2004, Electron. Mark..

[6]  Robin Hanson Idea Futures: Encouraging an Honest Consensus , 2013 .

[7]  Thavamalar Govindasamy,et al.  Successful implementation of e-Learning: Pedagogical considerations , 2001, Internet High. Educ..

[8]  Maja J. Matarić,et al.  Designing emergent behaviors: from local interactions to collective intelligence , 1993 .

[9]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[10]  Henry A. Kautz,et al.  Real-time crowd labeling for deployable activity recognition , 2013, CSCW.

[11]  Luis von Ahn Games with a Purpose , 2006, Computer.

[12]  Tatsuo Nakajima,et al.  Drawing on mobile crowds via social media , 2012, Multimedia Systems.

[13]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[14]  Lukas Biewald,et al.  Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing , 2011, Human Computation.

[15]  Clara Pereira Coutinho Individual versus collaborative computer-supported concept mapping: A study with adult learners , 2009 .

[16]  Udo Kruschwitz,et al.  Assessing Crowdsourcing Quality through Objective Tasks , 2012, LREC.

[17]  David R. Michael,et al.  Serious Games: Games That Educate, Train, and Inform , 2005 .

[18]  E. Haugen Dialect, Language, Nation1 , 1966 .

[19]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[20]  D. Garrison E-Learning in the 21st Century , 2016 .

[21]  Doug Schuler,et al.  Social computing , 1994, CACM.

[22]  Wendy S. Ark,et al.  At What Cost Pervasive? A Social Comuting View of Mobile Computing Systems , 1999, IBM Syst. J..

[23]  François Bry,et al.  Human Computation-Enabled Network Analysis for a Systemic Credit Risk Rating , 2013, Handbook of Human Computation.

[24]  Yiftach Nagar,et al.  Improving Predictions with Hybrid Markets , 2012, AAAI Fall Symposium: Machine Aggregation of Human Judgment.

[25]  Mark Steyvers,et al.  Identifying Emotions, Intentions, and Attitudes in Text Using a Game with a Purpose , 2010, HLT-NAACL 2010.

[26]  Wenji Mao,et al.  Social Computing: From Social Informatics to Social Intelligence , 2007, IEEE Intell. Syst..

[27]  François Bry,et al.  Karido: A GWAP for telling artworks apart , 2011, 2011 16th International Conference on Computer Games (CGAMES).

[28]  John Seely Brown,et al.  Intelligent Tutoring Systems , 2016, Lecture Notes in Computer Science.

[29]  Brendan T. O'Connor,et al.  A Latent Variable Model for Geographic Lexical Variation , 2010, EMNLP.

[30]  Paul C. Tetlock,et al.  The Promise of Prediction Markets , 2008, Science.

[31]  Andrew B. Whinston,et al.  Research Issues in Social computing , 2007, J. Assoc. Inf. Syst..

[32]  Christian Wagner,et al.  Weblogging: A study of social computing and its impact on organizations , 2008, Decis. Support Syst..

[33]  Mark S. Ackerman,et al.  Crowdsourcing and knowledge sharing: strategic user behavior on taskcn , 2008, EC '08.

[34]  Linda O. Valenty,et al.  Political leadership for the new century : personality and behavior among American leaders , 2002 .

[35]  Michael B. Abramowicz,et al.  Information Markets, Administrative Decisionmaking, and Predictive Cost-Benefit Analysis , 2003 .

[36]  Rachel Davies,et al.  Agile Coaching , 2009 .

[37]  Jirí Mírovský,et al.  Designing a Language Game for Collecting Coreference Annotation , 2009, Linguistic Annotation Workshop.

[38]  Thomas A. Rietz,et al.  Accuracy and Forecast Standard Error of Prediction Markets , 2003 .

[39]  Tatsuo Nakajima,et al.  Gamifying intelligent environments , 2011, Ubi-MUI '11.

[40]  Simon L. Peyton Jones,et al.  A history of Haskell: being lazy with class , 2007, HOPL.