Automatic construction of English/Chinese parallel corpora

As the demand for global information increases significantly, multilingual corpora has become a valuable linguistic resource for applications to cross-lingual information retrieval and natural language processing. In order to cross the boundaries that exist between different languages, dictionaries are the most typical tools. However, the general-purpose dictionary is less sensitive in both genre and domain. It is also impractical to manually construct tailored bilingual dictionaries or sophisticated multilingual thesauri for large applications. Corpus-based approaches, which do not have the limitation of dictionaries, provide a statistical translation model with which to cross the language boundary. There are many domain-specific parallel or comparable corpora that are employed in machine translation and cross-lingual information retrieval. Most of these are corpora between Indo-European languages, such as English/French and English/Spanish. The Asian/Indo-European corpus, especially English/Chinese corpus, is relatively sparse. The objective of the present research is to construct English/ Chinese parallel corpus automatically from the World Wide Web. In this paper, an alignment method is presented which is based on dynamic programming to identify the one-to-one Chinese and English title pairs. The method includes alignment at title level, word level and character level. The longest common subsequence (LCS) is applied to find the most reliable Chinese translation of an English word. As one word for a language may translate into two or more words repetitively in another language, the edit operation, deletion, is used to resolve redundancy. A score function is then proposed to determine the optimal title pairs. Experiments have been conducted to investigate the performance of the proposed method using the daily press release articles by the Hong Kong SAR government as the test bed. The precision of the result is 0.998 while the recall is 0.806. The release articles and speech articles, published by Hongkong & Shanghai Banking Corporation Limited, are also used to test our method, the precision is 1.00, and the recall is 0.948.

[1]  RetrievalDouglas W. OardCollege.  Alternative Approaches for Cross-Language Text Retrieval , 1997 .

[2]  Douglas W. Oard,et al.  A survey of multilingual text retrieval , 1996 .

[3]  Michel Simard,et al.  Using cognates to align sentences in bilingual corpora , 1993, TMI.

[4]  Trevor I. Dix,et al.  A Bit-String Longest-Common-Subsequence Algorithm , 1986, Inf. Process. Lett..

[5]  Georgios Paliouras,et al.  Using Machine Learning to Maintain Rule-based Named-Entity Recognition and Classification Systems , 2001, ACL.

[6]  Jean Véronis,et al.  Parallel text processing :alignment and use of translationcorpora , 2000 .

[7]  Philip Resnik,et al.  Mining the Web for Bilingual Text , 1999, ACL.

[8]  I. Dan Melamed,et al.  Automatic Evaluation and Uniform Filter Cascades for Inducing N-Best Translation Lexicons , 1995, VLC@ACL.

[9]  John D. Lafferty,et al.  The Candide System for Machine Translation , 1994, HLT.

[10]  Jian Cai,et al.  Filtering noisy parallel corpora of web pages , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[11]  R. K. Shyamasundar,et al.  Introduction to algorithms , 1996 .

[12]  Jarle Ebeling,et al.  Contrastive Linguistics, Translation, and Parallel Corpora , 1998 .

[13]  John Cocke,et al.  A Statistical Approach to Machine Translation , 1990, CL.

[14]  Federico Zanettin,et al.  Bilingual Comparable Corpora and the Training of Translators , 1998 .

[15]  Ernst-August Gutt.  Translation and Relevance: Cognition and Context , 1993 .

[16]  Mona Baker,et al.  In Other Words: A Coursebook on Translation , 1993 .

[17]  Kenneth Ward Church,et al.  Identifying Word Correspondences in Parallel Texts , 1991, HLT.

[18]  E. Nida.  Toward a science of translating , 1964 .

[19]  David Yarowsky,et al.  Inducing Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora , 2001, HLT.

[20]  I. Dan Melamed.  Automatic Discovery of Non-Compositional Compounds in Parallel Data , 1997, EMNLP.

[21]  Mirella Lapata,et al.  Using Subcategorization to Resolve Verb Class Ambiguity , 1999, EMNLP.

[22]  T. Salthouse,et al.  Cognition and context. , 1992, Science.

[23]  Xiaoyi Ma,et al.  BITS: a method for bilingual text search over the Web , 1999, MTSUMMIT.

[24]  Philip Resnik,et al.  Semi-Automatic Acquisition of Domain-Specific Translation Lexicons , 1997, ANLP.

[25]  Elliott Macklovitch,et al.  Line ‘Em Up: Advances in Alignment Technology and their Impact on Translation Support Tools , 2004, Machine Translation.

[26]  Francisco Casacuberta,et al.  Search algorithms for statistical machine translation based on dynamic programming and pruning techniques , 2001 .

[27]  Kenneth Ward Church,et al.  Identifying word correspondence in parallel texts , 1991 .

[28]  I. Dan Melamed,et al.  A Geometric Approach to Mapping Bitext Correspondence , 1996, EMNLP.

[29]  Evelyne Tzoukermann,et al.  The BICORD System Combining Lexical Information from Bilingual Corpora and Machine Readable Dictionaries , 1990, COLING.

[30]  Philip Resnik,et al.  Parallel strands: a preliminary investigation into mining the Web for bilingual text , 1998, AMTA.

[31]  Robert L. Mercer,et al.  Aligning Sentences in Parallel Corpora , 1991, ACL.

[32]  Mildred L. Larson,et al.  Meaning-Based Translation: A Guide to Cross-Language Equivalence , 1986 .

[33]  Kenneth Ward Church,et al.  Aligning Parallel Texts : Do Methods Developed for English-French Generalize to Asian Languages? , 1993 .

[34]  Jean Paul Ballerini,et al.  Experiments in multilingual information retrieval using the SPIDER system , 1996, SIGIR '96.

[35]  Jean Véronis,et al.  Parallel Text Processing , 2000 .

[36]  Katsumi Takahashi,et al.  Sentence Alignment in Parallel Corpora: The Asahi Corpus of Newspaper Editorials , 1995 .

[37]  Jian Zhang,et al.  On the use of words and n-grams for Chinese information retrieval , 2000, IRAL '00.

[38]  I. Dan Melamed,et al.  Manual Annotation of Translational Equivalence: The Blinker Project , 1998, ArXiv.

[39]  Pierre Isabelle,et al.  Word Completion- A First Step Toward Target-Text Mediated IMT , 1996, COLING.

[40]  Jian-Yun Nie,et al.  Automatic construction of parallel English-Chinese corpus for cross-language information retrieval , 2000, ANLP.

[41]  Gwyneth Tseng,et al.  Chinese text segmentation for text retrieval: achievements and problems , 1993 .

[42]  Michel Simard,et al.  Multialignement vs bialignement : à plusieurs, c’est mieux ! , 2015, JEPTALNRECITAL.

[43]  Jean V ronis.  Parallel Text Processing: Alignment and Use of Translation Corpora , 2002 .

[44]  Jennifer Widom,et al.  Change detection in hierarchically structured information , 1996, SIGMOD '96.

[45]  Hermann Ney,et al.  Improved Statistical Alignment Models , 2000, ACL.

[46]  Mark W. Davis,et al.  Query Translation Using Evolutionary Programming for Multilingual Information Retrieval II , 1995, Evolutionary Programming.

[47]  Xiaobo Ren,et al.  Translation Analysis and Translation Automation , 1993, TMI.

[48]  Paola Velardi,et al.  Automatic adaptation of proper noun dictionaries through cooperation of machine learning and probabilistic methods , 2000, SIGIR '00.

[49]  Shaoyi He.  Translingual alteration of conceptual information in medical translation , 2000 .

[50]  Guy Aston,et al.  Corpus Use and Learning to Translate , 1999 .

[51]  Carol Peters,et al.  Reference corpora and lexicons for translators and translation studies , 1997 .

[52]  Jian-Yun Nie,et al.  Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the Web , 1999, SIGIR '99.

[53]  Ernst-August Gutt,et al.  Translation and relevance , 1991 .

[54]  L. Allison,et al.  Fast, optimal alignment of three sequences using linear gap costs. , 2000, Journal of theoretical biology.

[55]  Jian-Yun Nie,et al.  On Chinese text retrieval , 1996, SIGIR '96.