Using semi-experts to derive judgments on word sense alignment: a pilot study

The overall goal of this project is to evaluate the performance of word sense alignment (WSA) systems, focusing on obtaining examples appropriate to language learners. Building a gold standard dataset based on human expert judgments is costly in time and labor, and thus we gauge the utility of using semi-experts in performing the annotation. In an online survey, we present a sense of a target word from one dictionary with senses from the other dictionary, asking for judgments of relatedness. We note the difficulty of agreement, yet the utility in using such results to evaluate WSA work. We find that one's treatment of related senses heavily impacts the results for WSA.

[1]  Nitin Madnani,et al.  They Can Help: Using Crowdsourcing to Improve the Evaluation of Grammatical Error Detection Systems , 2011, ACL.

[2]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[3]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[4]  Roberto Navigli.  Reducing the Granularity of a Computational Lexicon via an Automatic Mapping to a Coarse-Grained Sense Inventory , 2006, LREC.

[5]  R. Navigli,et al.  SemEval-2007 Task 07: Coarse-Grained English All-Words Task , 2007, International Workshop on Semantic Evaluation.

[6]  John Nerbonne,et al.  GLOSSER-RuG: in Support of Reading , 1996, COLING.

[7]  Iryna Gurevych,et al.  Aligning Sense Inventories in Wikipedia and WordNet , 2010 .

[8]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[9]  Vivi Nastase.  Word Sense Disambiguation in Roget's Thesaurus Using WordNet , 2012 .

[10]  Martha Palmer,et al.  Sense Tagging the Penn Treebank , 2000 .

[11]  Maxine Eskénazi,et al.  Classroom success of an intelligent tutoring system for lexical practice and reading comprehension , 2006, INTERSPEECH.

[12]  Min-Yen Kan,et al.  Perspectives on crowdsourcing annotations for natural language processing , 2012, Language Resources and Evaluation.

[13]  Mark Stevenson,et al.  Aligning WordNet Synsets and Wikipedia Articles , 2010, Collaboratively-Built Knowledge Sources and AI.

[14]  Maria Ruiz-Casado,et al.  Automatic Assignment of Wikipedia Encyclopedic Entries to WordNet Synsets , 2005, AWIC.

[15]  Katrin Erk,et al.  Graded Word Sense Assignment , 2009, EMNLP.

[16]  Iryna Gurevych,et al.  What Psycholinguists Know About Chemistry: Aligning Wiktionary and WordNet for Increased Domain Coverage , 2011, IJCNLP.

[17]  Simone Paolo Ponzetto,et al.  BabelNet: Building a Very Large Multilingual Semantic Network , 2010, ACL.

[18]  Iryna Gurevych,et al.  The People’s Web meets Linguistic Knowledge: Automatic Sense Alignment of Wikipedia and WordNet , 2011, IWCS.

[19]  Roberto Navigli,et al.  Word sense disambiguation: A survey , 2009, CSUR.

[20]  Jean Véronis,et al.  Mapping Dictionaries: A Spreading Activation Approach , 1999 .

[21]  Ted Pedersen,et al.  WordNet::SenseRelate::AllWords - A Broad Coverage Word Sense Tagger that Maximizes Semantic Relatedness , 2009, NAACL.

[22]  Kevin Knight,et al.  Building a Large-Scale Knowledge Base for Machine Translation , 1994, AAAI.

[23]  Kristiina Muhonen,et al.  Creating a Dependency Syntactic Treebank: Towards Intuitive Language Modeling , 2011 .

[24]  Hinrich Schütze,et al.  Active Learning with Amazon Mechanical Turk , 2011, EMNLP.

[25]  Oi Yee Kwong.  Aligning WordNet with Additional Lexical Resources , 1998, WordNet@ACL/COLING.

[26]  Katrin Erk,et al.  Investigations on Word Senses and Word Usages , 2009, ACL.