Multilingual domain modeling in Twenty-One: automatic creation of a bi-directional translation lexicon from a parallel corpus

Within the project Twenty-One, which aims at the effective dissemination of information on ecology and sustainable development, a sytem is developed that supports cross-language information retrieval in any of the four languages Dutch, English, French and German. Knowledge of this application domain is needed to enhance existing translation resources for the purpose of lexical disambiguation. This paper describes an algorithm for the automated acquisition of a translation lexicon from a parallel corpus. New about the presented algorithm is the statistical language model used. Because the algorithm is based on a symmetric translation model it becomes possible to identify one-to-many and many-to-one relations between words of a language pair. We claim that the presented method has two advantages over algorithms that have been published before. Firstly, because the translation model is more powerful, the resulting bilingual lexicon will be more accurate. Secondly, the resulting bilingual lexicon can be used to translate in both directions between a language pair. Different versions of the algorithm were evaluated on the Dutch and English version of the Agenda 21 corpus, which is a UN document on the application domain of sustainable development.

[1]  Wessel Kraaij,et al.  Multilingual functionality in the TwentyOne project , 1997 .

[2]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[3]  Kenneth Ward Church,et al.  A Program for Aligning Sentences in Bilingual Corpora , 1993, CL.

[4]  Djoerd Hiemstra,et al.  A domain Specific Lexicon Acquisition Tool for Cross-Language Information Retrieval , 1997, RIAO.

[5]  I. Dan Melamed.  Automatic Discovery of Non-Compositional Compounds in Parallel Data , 1997, EMNLP.

[6]  Wessel Kraaij,et al.  Twenty-One: Cross-Language Disclosure and Retrieval of Multimedia Documents on Sustainable Development , 1998, Comput. Networks.

[7]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[8]  Nils J. Nilsson,et al.  Principles of Artificial Intelligence , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Pim van der Eijk.  Automating the Acquisition of Bilingual Terminology , 1993, EACL.

[10]  Kenneth Ward Church,et al.  Identifying word correspondence in parallel texts , 1991 .

[11]  I. Dan Melamed.  A Word-to-Word Model of Translational Equivalence , 1997, ACL.

[12]  Vasileios Hatzivassiloglou,et al.  Translating Collocations for Bilingual Lexicons: A Statistical Approach , 1996, CL.

[13]  John Cocke,et al.  A Statistical Approach to Machine Translation , 1990, CL.

[14]  Julian Kupiec,et al.  An Algorithm for Finding Noun Phrase Correspondences in Bilingual Corpora , 1993, ACL.

[15]  Djoerd Hiemstra.  Deriving a Bilingual Lexicon for Cross-Language Information Retrieval , 1997 .

[16]  Kenneth Ward Church,et al.  Identifying Word Correspondences in Parallel Texts , 1991, HLT.

[17]  PietraVincent J. Della,et al.  The mathematics of statistical machine translation , 1993 .