A formal model of ambiguity and its applications in machine translation

Systems that process natural language must cope with and resolve ambiguity. In this dissertation, a model of language processing is advocated in which multiple inputs and multiple analyses of inputs are considered concurrently and a single analysis is only a last resort. Compared to conventional models, this approach can be understood as replacing single-element inputs and outputs with weighted sets of inputs and outputs. Although processing components must deal with sets (rather than individual elements), constraints are imposed on the elements of these sets, and the representations from existing models may be reused. However, to deal efficiently with large (or infinite) sets, compact representations of sets that share structure between elements, such as weighted finite-state transducers and synchronous context-free grammars, are necessary. These representations and algorithms for manipulating them are discussed in depth in depth. To establish the effectiveness and tractability of the proposed processing model, it is applied to several problems in machine translation. Starting with spoken language translation, it is shown that translating a set of transcription hypotheses yields better translations compared to a baseline in which a single (1-best) transcription hypothesis is selected and then translated, independent of the translation model formalism used. More subtle forms of ambiguity that arise even in text-only translation (such as decisions conventionally made during system development about how to preprocess text) are then discussed, and it is shown that the ambiguity-preserving paradigm can be employed in these cases as well, again leading to improved translation quality. A model for supervised learning that learns from training data where sets (rather than single elements) of correct labels are provided for each training instance and use it to learn a model of compound word segmentation is also introduced, which is used as a preprocessing step in machine translation.

[1]  Jason Eisner,et al.  Directional Constraint Evaluation in Optimality Theory , 2022 .

[2]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[3]  Giorgio Gallo,et al.  Directed Hypergraphs and Applications , 1993, Discret. Appl. Math..

[4]  Jimmy J. Lin,et al.  Book Reviews: Data-Intensive Text Processing with MapReduce by Jimmy Lin and Chris Dyer , 2010, CL.

[5]  Stefan Ortmanns,et al.  High quality word graphs using forward-backward pruning , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[6]  John DeNero,et al.  Sampling Alignment Structure under a Bayesian Translation Model , 2008, EMNLP.

[7]  Richard Sproat,et al.  The First International Chinese Word Segmentation Bakeoff , 2003, SIGHAN.

[8]  Sergey Bratus,et al.  Experiments in Multi-Modal Automatic Content Extraction , 2001, HLT.

[9]  I. Dan Melamed,et al.  Statistical Machine Translation by Parsing , 2004, ACL.

[10]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[11]  Andreas Stolcke,et al.  SRILM - an extensible language modeling toolkit , 2002, INTERSPEECH.

[12]  Clifford J. Weinstein,et al.  Measuring Translation Quality by Testing English Speakers with a New Defense Language Proficiency Test for Arabic , 2005 .

[13]  Christopher J. Dyer,et al.  The “Noisier Channel”: Translation from Morphologically Complex Languages , 2007, WMT@ACL.

[14]  Andrew Y. Ng,et al.  Solving the Problem of Cascading Errors: Approximate Bayesian Inference for Linguistic Annotation Pipelines , 2006, EMNLP.

[15]  Jason Eisner,et al.  Learning Linear Ordering Problems for Better Translation , 2009, EMNLP.

[16]  Hermann Ney,et al.  Morpho-syntactic analysis for reordering in statistical machine translation , 2001, MTSUMMIT.

[17]  Daniel Gildea,et al.  Unsupervised Tokenization for Machine Translation , 2009, EMNLP.

[18]  Shankar Kumar,et al.  Minimum Bayes-Risk Decoding for Statistical Machine Translation , 2004, NAACL.

[19]  Haizhou Li,et al.  Topological Ordering of Function Words in Hierarchical Phrase-based Translation , 2009, ACL/IJCNLP.

[20]  Giorgio Satta,et al.  The language intersection problem for non-recursive context-free grammars , 2004, Inf. Comput..

[21]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[22]  Shankar Kumar,et al.  Lattice Minimum Bayes-Risk Decoding for Statistical Machine Translation , 2008, EMNLP.

[23]  David Chiang,et al.  Forest Rescoring: Faster Decoding with Integrated Language Models , 2007, ACL.

[24]  Zhifei Li,et al.  First- and Second-Order Expectation Semirings with Applications to Minimum-Risk Training on Translation Forests , 2009, EMNLP.

[25]  Mark Johnson,et al.  Nonparametric bayesian models of lexical acquisition , 2007 .

[26]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[27]  Noah A. Smith,et al.  Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction , 2009, NAACL.

[28]  Koby Crammer,et al.  Sequence Learning from Data with Multiple Labels , 2009 .

[29]  Christopher D. Manning,et al.  Accurate Non-Hierarchical Phrase-Based Translation , 2010, NAACL.

[30]  Arto Salomaa,et al.  Semirings, Automata, Languages , 1985, EATCS Monographs on Theoretical Computer Science.

[31]  Sharon Goldwater,et al.  Improving Statistical MT through Morphological Analysis , 2005, HLT.

[32]  Ben Taskar,et al.  Posterior Regularization for Structured Latent Variable Models , 2010, J. Mach. Learn. Res..

[33]  Franz Josef Och,et al.  Minimum Error Rate Training in Statistical Machine Translation , 2003, ACL.

[34]  B. Partee,et al.  Mathematical Methods in Linguistics , 1990 .

[35]  Jan Hajic,et al.  Tagging Inflective Languages: Prediction of Morphological Categories for a Rich Structured Tagset , 1998, ACL.

[36]  Hermann Ney,et al.  Discriminative Reordering Models for Statistical Machine Translation , 2006, WMT@HLT-NAACL.

[37]  Mark Johnson,et al.  Improving Unsupervised Dependency Parsing with Richer Contexts and Smoothing , 2009, NAACL.

[38]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[39]  Philipp Koehn,et al.  Findings of the 2009 Workshop on Statistical Machine Translation , 2009, WMT@EACL.

[40]  Kenneth Ward Church,et al.  Good applications for crummy machine translation , 1993, Machine Translation.

[41]  Christopher D. Manning,et al.  Optimizing Chinese Word Segmentation for Machine Translation Performance , 2008, WMT@ACL.

[42]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[43]  Lillian Lee,et al.  Fast context-free grammar parsing requires fast boolean matrix multiplication , 2001, JACM.

[44]  Masao Utiyama,et al.  Paraphrase Lattice for Statistical Machine Translation , 2010, ACL.

[45]  Pietro Perona,et al.  Inferring Ground Truth from Subjective Labelling of Venus Images , 1994, NIPS.

[46]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[47]  Daniel Marcu,et al.  Statistical Phrase-Based Translation , 2003, NAACL.

[48]  Miles Osborne,et al.  Modelling Lexical Redundancy for Machine Translation , 2006, ACL.

[49]  Nizar Habash,et al.  Permission is granted to quote short excerpts and to reproduce figures and tables from this report, provided that the source of such material is fully acknowledged. Arabic Preprocessing Schemes for Statistical Machine Translation , 2006 .

[50]  Andreas Stolcke,et al.  Finding consensus in speech recognition: word error minimization and other applications of confusion networks , 2000, Comput. Speech Lang..

[51]  Philip Koehn,et al.  Statistical Machine Translation , 2010, EAMT.

[52]  Phil Blunsom,et al.  A Discriminative Latent Variable Model for Statistical Machine Translation , 2008, ACL.

[53]  Sanjeev Khudanpur,et al.  Variational Decoding for Statistical Machine Translation , 2009, ACL.

[54]  Fernando Pereira,et al.  Shallow Parsing with Conditional Random Fields , 2003, NAACL.

[55]  Dan Klein,et al.  Discriminative Log-Linear Grammars with Latent Variables , 2007, NIPS.

[56]  Brian Roark,et al.  Generalized Algorithms for Constructing Statistical Language Models , 2003, ACL.

[57]  Mark-Jan Nederhof,et al.  Squibs and Discussions: Weighted Deductive Parsing and Knuth’s Algorithm , 2003, CL.

[58]  Ralph Weischedel,et al.  A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION , 2005 .

[59]  Geoffrey Zweig,et al.  The IBM 2006 Gale Arabic ASR System , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[60]  Hermann Ney,et al.  Do We Need Chinese Word Segmentation for Statistical Machine Translation? , 2004, SIGHAN@ACL.

[61]  Martin Rajman,et al.  Lattice Parsing for Speech Recognition , 1999 .

[62]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[63]  Stanley F. Chen,et al.  An empirical study of smoothing techniques for language modeling , 1999 .

[64]  Daniel Jurafsky,et al.  Disambiguating “DE” for Chinese-English Machine Translation , 2009, WMT@EACL.

[65]  Hermann Ney,et al.  A Systematic Comparison of Various Statistical Alignment Models , 2003, CL.

[66]  Liang Huang,et al.  Forest-based algorithms in natural language processing , 2008 .

[67]  Yanjun Ma,et al.  Bootstrapping Word Alignment via Word Packing , 2007, ACL.

[68]  William W. Cohen,et al.  Semi-Markov Conditional Random Fields for Information Extraction , 2004, NIPS.

[69]  Giorgio Satta,et al.  Probabilistic Parsing as Intersection , 2003, IWPT.

[70]  Giorgio Satta,et al.  Some Computational Complexity Results for Synchronous Context-Free Grammars , 2005, HLT/EMNLP.

[71]  Peng Xu,et al.  Using a Dependency Parser to Improve SMT for Subject-Object-Verb Languages , 2009, NAACL.

[72]  Cyril Allauzen,et al.  3-Way Composition of Weighted Finite-State Transducers , 2008, CIAA.

[73]  Stephan Vogel,et al.  An Efficient Two-Pass Approach to Synchronous-CFG Driven Statistical MT , 2007, NAACL.

[74]  J. Fodor,et al.  The Modularity of Mind: An Essay on Faculty Psychology , 1984 .

[75]  Chris Dyer,et al.  Using a maximum entropy model to build segmentation lattices for MT , 2009, NAACL.

[76]  Dekai Wu,et al.  Stochastic Inversion Transduction Grammars and Bilingual Parsing of Parallel Corpora , 1997, CL.

[77]  K. Vijay-Shanker,et al.  The Use of Shared Forests in Tree Adjoining Grammar Parsing , 1993, EACL.

[78]  Nitin Madnani,et al.  Generating Phrasal and Sentential Paraphrases: A Survey of Data-Driven Methods , 2010, CL.

[79]  Peter Burke,et al.  Log-linear models , 1980 .

[80]  Chao Wang,et al.  Chinese Syntactic Reordering for Statistical Machine Translation , 2007, EMNLP.

[81]  Mehryar Mohri,et al.  Weighted Automata Algorithms , 2009 .

[82]  Zhao Tie.  Increasing Accuracy of Chinese Segmentation with Strategy of Multi step Processing , 2001 .

[83]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[84]  Joshua Goodman,et al.  Semiring Parsing , 1999, CL.

[85]  Olivia Buzek,et al.  Improving Translation via Targeted Paraphrasing , 2010, EMNLP.

[86]  Sanjeev Khudanpur,et al.  Efficient Extraction of Oracle-best Translations from Hypergraphs , 2009, HLT-NAACL.

[87]  Shankar Kumar,et al.  A weighted finite state transducer translation template model for statistical machine translation , 2006, Nat. Lang. Eng..

[88]  Lauri Karttunen,et al.  Finite State Morphology , 2003, CSLI Studies in Computational Linguistics.

[89]  James F. Allen.  Natural language understanding , 1987, Bejnamin/Cummings series in computer science.

[90]  Sanjeev Khudanpur,et al.  A Scalable Decoder for Parsing-Based Machine Translation with Equivalent Language Model State Maintenance , 2008, SSST@ACL.

[91]  Razvan C. Bunescu.  Learning with Probabilistic Features for Improved Pipeline Models , 2008, EMNLP.

[92]  Charles Kemp,et al.  Bayesian models of cognition , 2008 .

[93]  Nizar Habash,et al.  Determining Case in Arabic: Learning Complex Linguistic Behavior Requires Complex Linguistic Features , 2007, EMNLP.

[94]  Bonnie J. Dorr,et al.  Interlingual Machine Translation: A Parameterized Approach , 1993, Artif. Intell..

[95]  Luis von Ahn.  Games with a Purpose , 2006, Computer.

[96]  Andreas Zollmann,et al.  Syntax Augmented Machine Translation via Chart Parsing , 2006, WMT@HLT-NAACL.

[97]  Philipp Koehn,et al.  Moses: Open Source Toolkit for Statistical Machine Translation , 2007, ACL.

[98]  Andrew McCallum,et al.  Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data , 2004, J. Mach. Learn. Res..

[99]  Kevin Knight,et al.  Synchronous Tree Adjoining Machine Translation , 2009, EMNLP.

[100]  Xu Sun,et al.  A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information , 2009, HLT-NAACL.

[101]  Rebecca N. Wright,et al.  Finite-State Approximation of Phrase Structure Grammars , 1991, ACL.

[102]  Philip Resnik,et al.  Context-free reordering, finite-state translation , 2010, HLT-NAACL.

[103]  Kevin Knight,et al.  Training Tree Transducers , 2004, NAACL.

[104]  Franz Josef Och,et al.  A Systematic Comparison of Phrase-Based, Hierarchical and Syntax-Augmented Statistical MT , 2008, COLING.

[105]  M. Droste,et al.  Semirings and Formal Power Series , 2009 .

[106]  David Chiang,et al.  Hierarchical Phrase-Based Translation , 2007, CL.

[107]  Hermann Ney,et al.  Generation of Word Graphs in Statistical Machine Translation , 2002, EMNLP.

[108]  Alexander M. Rush,et al.  On Dual Decomposition and Linear Programming Relaxations for Natural Language Processing , 2010, EMNLP.

[109]  Mei Yang,et al.  Phrase-Based Backoff Models for Machine Translation of Highly Inflected Languages , 2006, EACL.

[110]  Shankar Kumar,et al.  Efficient Minimum Error Rate Training and Minimum Bayes-Risk Decoding for Translation Hypergraphs and Lattices , 2009, ACL/IJCNLP.

[111]  Richard Edwin Stearns,et al.  Syntax-Directed Transduction , 1966, JACM.

[112]  Hermann Ney,et al.  Improved backing-off for M-gram language modeling , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[113]  Christos H. Papadimitriou,et al.  Elements of the Theory of Computation , 1997, SIGA.

[114]  Dan Klein,et al.  Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency , 2004, ACL.

[115]  Michael Collins,et al.  A New Statistical Parser Based on Bigram Lexical Dependencies , 1996, ACL.

[116]  Adam Lopez,et al.  Translation as Weighted Deduction , 2009, EACL.

[117]  Alexandra Birch,et al.  A Quantitative Analysis of Reordering Phenomena , 2009, WMT@EACL.

[118]  Eiichiro Sumita,et al.  Toward a Broad-coverage Bilingual Corpus for Speech Translation of Travel Conversations in the Real World , 2002, LREC.

[119]  Chris Dyer,et al.  A Gibbs Sampler for Phrasal Synchronous Grammar Induction , 2009, ACL.

[120]  Nitin Madnani,et al.  The circle of meaning: from translation to paraphrasing and back , 2010 .

[121]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[122]  Steven J. DeRose,et al.  Grammatical Category Disambiguation by Statistical Optimization , 1988, CL.

[123]  Pierre Boullier,et al.  Range Concatenation Grammars , 2000, IWPT.

[124]  Aravind K. Joshi,et al.  Using Lexicalized Tags for Machine Translation , 1990, COLING.

[125]  Mark de Berg,et al.  Computational geometry: algorithms and applications , 1997 .

[126]  David Chiang,et al.  Better k-best Parsing , 2005, IWPT.

[127]  Adam David Lopez,et al.  Machine Translation by Pattern Matching , 2008 .

[128]  Jay Earley,et al.  An efficient context-free parsing algorithm , 1970, Commun. ACM.

[129]  Kevin Knight,et al.  A Syntax-based Statistical Translation Model , 2001, ACL.

[130]  Kemal Oflazer,et al.  Exploring Different Representational Units in English-to-Turkish Statistical Machine Translation , 2007, WMT@ACL.

[131]  Roger L. Freeman.  Wiley Series in Telecommunications and Signal Processing , 2005 .

[132]  Richard Zens,et al.  Speech Translation by Confusion Network Decoding , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[133]  Rong Jin,et al.  Learning with Multiple Labels , 2002, NIPS.

[134]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[135]  Daniel Gildea.  Parsers as language models for statistical machine translation , 2008 .

[136]  Chris Callison-Burch,et al.  Demonstration of Joshua: An Open Source Toolkit for Parsing-based Machine Translation , 2009, ACL.

[137]  Ceriel J. H. Jacobs,et al.  Parsing as Intersection , 2008 .

[138]  Xavier Carreras,et al.  Non-Projective Parsing for Statistical Machine Translation , 2009, EMNLP.

[139]  John DeNero,et al.  Better Word Alignments with Supervised ITG Models , 2009, ACL.

[140]  Trevor Darrell,et al.  Conditional Random Fields for Object Recognition , 2004, NIPS.

[141]  Ossama Emam,et al.  Language Model Based Arabic Word Segmentation , 2003, ACL.

[142]  Philipp Koehn,et al.  Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models , 2004, AMTA.

[143]  James R. Curran,et al.  Parsing the WSJ Using CCG and Log-Linear Models , 2004, ACL.

[144]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[145]  Philipp Koehn,et al.  Clause Restructuring for Statistical Machine Translation , 2005, ACL.

[146]  Smaranda Muresan,et al.  Generalizing Word Lattice Translation , 2008, ACL.

[147]  Mark Steedman,et al.  The syntactic process , 2004, Language, speech, and communication.

[148]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[149]  Chris Callison-Burch,et al.  Feasibility of Human-in-the-loop Minimum Error Rate Training , 2009, EMNLP.

[150]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[151]  E. Jaynes.  Probability theory : the logic of science , 2003 .

[152]  William J. Byrne,et al.  Statistical Phrase-Based Speech Translation , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[153]  Xiaoqiang Luo,et al.  A Statistical Model for Multilingual Entity Detection and Tracking , 2004, NAACL.

[154]  Daniel Gildea,et al.  Binarization of Synchronous Context-Free Grammars , 2009, CL.

[155]  Chris Dyer.  Two monolingual parses are better than one (synchronous parse) , 2010, HLT-NAACL.

[156]  Daniel Gildea,et al.  Bayesian Learning of Non-Compositional Phrases with Synchronous Parsing , 2008, ACL.

[157]  Philipp Koehn,et al.  Empirical Methods for Compound Splitting , 2003, EACL.

[158]  Janyce Wiebe,et al.  Development and Use of a Gold-Standard Data Set for Subjectivity Classifications , 1999, ACL.

[159]  Johan Schalkwyk,et al.  OpenFst: A General and Efficient Weighted Finite-State Transducer Library , 2007, CIAA.

[160]  Philipp Koehn,et al.  Enabling Monolingual Translators: Post-Editing vs. Options , 2010, NAACL.

[161]  Christopher Dyer.  The University of maryland translation system for IWSLT 2007 , 2007, IWSLT.

[162]  G. Elisabeta Marai,et al.  The Chinese Room: Visualization and Interaction to Understand and Correct Ambiguous Machine Translation , 2009, Comput. Graph. Forum.

[163]  Galen Andrew,et al.  A Hybrid Markov/Semi-Markov Conditional Random Field for Sequence Segmentation , 2006, EMNLP.

[164]  Philipp Koehn.  Introduction to statistical machine translation , 2004, AMTA.

[165]  Michael Collins,et al.  Hidden-Variable Models for Discriminative Reranking , 2005, HLT.

[166]  Philipp Koehn,et al.  Towards better Machine Translation Quality for the German-English Language Pairs , 2008, WMT@ACL.

[167]  Wolfgang Macherey,et al.  Lattice-based Minimum Error Rate Training for Statistical Machine Translation , 2008, EMNLP.

[168]  Walter L. Ruzzo,et al.  An Improved Context-Free Recognizer , 1980, ACM Trans. Program. Lang. Syst..