Morpho Challenge Evaluation by Information Retrieval Experiments

In Morpho Challenge competitions, the objective has been to design statistical machine learning algorithms that discover which morphemes (smallest individually meaningful units of language) words consist of. Ideally, these are basic vocabulary units suitable for different tasks, such as text understanding, machine translation, information retrieval (IR), and statistical language modeling. In this paper, we propose to evaluate the morpheme analyses by performing IR experiments, where the words in the documents and queries are replaced by their proposed morpheme representations and the search is based on morphemes instead of words. In this paper, the evaluations are run for three languages: Finnish, German, and English using the queries, texts, and relevance judgments available in CLEF forum. The results show that the morpheme analysis has a significant effect in IR performance in all languages, and that the performance of the best unsupervised methods can be superior to the supervised reference methods.

[1]  Carol Peters,et al.  Advances in Multilingual and Multimodal Information Retrieval, 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers , 2008, CLEF.

[2]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[3]  Mathias Creutz,et al.  Unsupervised Discovery of Morphemes , 2002, SIGMORPHON.

[4]  Jeff A. Bilmes,et al.  Factored Language Models and Generalized Parallel Backoff , 2003, NAACL.

[5]  Fei Xia,et al.  A Hybrid Approach to the Induction of Underlying Morphology , 2008, IJCNLP.

[6]  Howard L. Bleich,et al.  Conceptual mapping of user's queries to medical subject headings , 1997, AMIA.

[7]  Julia Hirschberg,et al.  HLT-NAACL 2004 : Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics : proceedings of the main conference : May 2-7, 2004, Boston, Massachusetts, USA , 2004 .

[8]  Mathias Creutz,et al.  Morpheme Segmentation Gold Standards for Finnish and English , 2004 .

[9]  Young-Suk Lee,et al.  Morphological Analysis for Statistical Machine Translation , 2004, NAACL.

[10]  Mikko Kurimo,et al.  Morpho Challenge Evaluation Using a Linguistic Gold Standard , 2007, CLEF.

[11]  Ebru Arisoy,et al.  Unsupervised segmentation of words into morphemes - Challenge 2005, An Introduction and Evaluation Report , 2006 .

[12]  M. F. Porter,et al.  An algorithm for suffix stripping , 1997 .

[13]  Mikko Kurimo,et al.  Unsupervised Morpheme Analysis Evaluation by IR experiments - Morpho Challenge 2007 , 2007, CLEF.

[14]  Mathias Creutz,et al.  INDUCING THE MORPHOLOGICAL LEXICON OF A NATURAL LANGUAGE FROM UNANNOTATED TEXT , 2005 .