Unsupervised grammar induction using history based approach

Grammar induction, also known as grammar inference, is one of the most important research areas in the domain of natural language processing. Availability of large corpora has encouraged many researchers to use statistical methods for grammar induction. This problem can be divided into three different categories of supervised, semi-supervised, and unsupervised, based on type of the required data set for the training phase. Most current inductive methods are supervised, which need a bracketed data set for their training phase; but the lack of this kind of data set in many languages, encouraged us to focus on unsupervised approaches. Here, we introduce a novel approach, which we call history-based inside-outside (HIO), for unsupervised grammar inference, by using part-of-speech tag sequences as the only source of lexical information. HIO is an extension of the inside-outside algorithm enriched by using some notions of history based approaches. Our experiments on English and Persian languages show that by adding some conditions to the rule assumptions of the induced grammar, one can achieve acceptable improvement in the quality of the output grammar.

[1]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[2]  John D. Lafferty,et al.  Development and Evaluation of a Broad-Coverage Probabilistic Grammar of English-Language Computer Manuals , 1992, ACL.

[3]  Mitchell P. Marcus,et al.  Parsing a Natural Language Using Mutual Information Statistics , 1990, AAAI.

[4]  Steve Young,et al.  Applications of stochastic context-free grammars using the Inside-Outside algorithm , 1990 .

[5]  Kenneth Ward Church.  A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text , 1988, ANLP.

[6]  Ceriel J. H. Jacobs,et al.  Parsing Techniques - A Practical Guide , 2007, Monographs in Computer Science.

[7]  Joan-Andreu Sánchez,et al.  Estimation of the probability distributions of stochastic context-free grammars from the k-best derivations , 1998, ICSLP.

[8]  John D. Lafferty,et al.  Decision Tree Parsing using a Hidden Derivation Model , 1994, HLT.

[9]  Roger K. Moore.  Computer Speech and Language , 1986 .

[10]  Tadao Kasami,et al.  An Efficient Recognition and Syntax-Analysis Algorithm for Context-Free Languages , 1965 .

[11]  Daniel H. Younger,et al.  Recognition and Parsing of Context-Free Languages in Time n^3 , 1967, Inf. Control..

[12]  Yves Schabes,et al.  Parsing the Wall Street Journal with the Inside-Outside Algorithm , 1993, EACL.

[13]  J. Baker.  Trainable grammars for speech recognition , 1979 .

[14]  Mark Johnson.  The effect of alternative tree epresentatmns on tree bank grammars , 1998, CoNLL.

[15]  Ralph Grishman,et al.  A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars , 1991, HLT.

[16]  Andreas Stolcke,et al.  Inducing Probabilistic Grammars by Bayesian Model Merging , 1994, ICGI.

[17]  Eric Brill,et al.  A corpus-based approach to language learning , 1993 .

[18]  Wendy J. Holmes,et al.  Speech Synthesis and Recognition , 1988 .

[19]  Vladimir Solmon,et al.  The estimation of stochastic context-free grammars using the Inside-Outside algorithm , 2003 .

[20]  José-Miguel Benedí,et al.  Probabilistic Estimation Of Stochastic Context-Free Grammars From The K-Best Derivations , 1999 .

[21]  Dan Klein,et al.  Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency , 2004, ACL.

[22]  Carl de Marcken,et al.  Unsupervised language acquisition , 1996, ArXiv.

[23]  Michael Collins,et al.  A New Statistical Parser Based on Bigram Lexical Dependencies , 1996, ACL.

[24]  Deniz Yuret,et al.  Discovery of linguistic relations using lexical attraction , 1998, ArXiv.

[25]  Mark A. Paskin,et al.  Grammatical Bigrams , 2001, NIPS.

[26]  Alexander Clark,et al.  Unsupervised Language Acquisition: Theory and Practice , 2002, ArXiv.

[27]  Menno van Zaanen.  ABL: Alignment-Based Learning , 2000, COLING.

[28]  Dan Klein,et al.  Natural Language Grammar Induction Using a Constituent-Context Model , 2001, NIPS.

[29]  David M. Magerman.  Statistical Decision-Tree Models for Parsing , 1995, ACL.

[30]  Alexander Clark.  Unsupervised induction of stochastic context-free grammars using distributional clustering , 2001, CoNLL.

[31]  Rémi Zajac,et al.  Persian-English Machine Translation: An Overview of the Shiraz Project , 2000 .

[32]  Dan Klein,et al.  Distributional phrase structure induction , 2001, CoNLL.

[33]  Heshaam Faili,et al.  An Application of Lexicalized Grammars in English-Persian Translation , 2004, ECAI.

[34]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[35]  Fernando Pereira,et al.  Inside-Outside Reestimation From Partially Bracketed Corpora , 1992, HLT.

[36]  Karine Megerdoomian,et al.  Persian Computational Morphology: A Unification-Based Approach , 2000 .

[37]  Stanley F. Chen,et al.  Bayesian Grammar Induction for Language Modeling , 1995, ACL.

[38]  F.,et al.  Learning of Stochastic Context-free Grammars from Bracketed Corpora by Means of Reestimation Algorithms , 1999 .

[39]  Menno van Zaanen,et al.  ABL: Alignment-Based Learning , 2000, COLING.

[40]  Francisco Casacuberta,et al.  Comparison Between the Inside-Outside Algorithm and the Viterbi Algorithm for Stochastic Context-Free Grammars , 1996, SSPR.

[41]  John D. Lafferty,et al.  Towards History-based Grammars: Using Richer Models for Probabilistic Parsing , 1993, ACL.

[42]  Christopher D. Manning,et al.  The unsupervised learning of natural language structure , 2005 .

[43]  Dan Klein,et al.  A Generative Constituent-Context Model for Improved Grammar Induction , 2002, ACL.

[44]  Glenn Carroll,et al.  Two Experiments on Learning Probabilistic Dependency Grammars from Corpora , 1992 .

[45]  Michael Collins,et al.  Three Generative, Lexicalised Models for Statistical Parsing , 1997, ACL.

[46]  Eugene Charniak,et al.  Statistical Parsing with a Context-Free Grammar and Word Statistics , 1997, AAAI/IAAI.

[47]  Menno van Zaanen,et al.  Comparing Two Unsupervised Grammar Induction Systems: Alignment-Based Learning vs. EMILE , 2001 .

[48]  Eugene Charniak,et al.  Statistical language learning , 1997 .

[49]  David M. Magerman,et al.  Efficiency, Robustness and Accuracy in Picky Chart Parsing , 1992, ACL.

[50]  Malcolm J. Bowman,et al.  Proceedings of the Workshop , 1978 .

[51]  Ted Briscoe,et al.  Robust stochastic parsing using the inside-outside algorithm , 1994, ArXiv.

[52]  Menno van Zaanen,et al.  Bootstrapping structure into language : alignment-based learning , 2001, ArXiv.

[53]  George R. Doddington,et al.  The ATIS Spoken Language Systems Pilot Corpus , 1990, HLT.

[54]  Mitchell P. Marcus,et al.  Pearl: A Probabilistic Chart Parser , 1991, EACL.

[55]  Eugene Charniak,et al.  A Maximum-Entropy-Inspired Parser , 2000, ANLP.

[56]  Pieter W. Adriaans,et al.  Grammar Induction as Substructural Inductive Logic Programming , 2001, Learning Language in Logic.

[57]  Eugene Charniak,et al.  Statistical Techniques for Natural Language Parsing , 1997, AI Mag..

[58]  James H. Martin,et al.  Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition , 2000 .

[59]  C. de Marcken.  On the Unsupervised Induction of Phrase-Structure Grammars , 1999 .

[60]  Francisco Casacuberta.  Growth Transformations for Probability Functions of Stochastic Grammars , 1996, Int. J. Pattern Recognit. Artif. Intell..