Rich Syntax from a Raw Corpus: Unsupervised Does It

We compare our model of unsupervised learning of linguistic structures, ADIOS [1], to some recent work in computational linguistics and in grammar theory. Our approach resembles the Construction Grammar in its general philosophy (e.g., in its reliance on structural generalizations rather than on syntax projected by the lexicon, as in the current generative theories), and the Tree Adjoining Grammar in its computational characteristics (e.g., in its apparent affinity with Mildly Context Sensitive Languages). The representations learned by our algorithm are truly emergent from the (unannotated) corpus data, whereas those found in published works on cognitive and construction grammars and on TAGs are hand-tailored. Thus, our results complement and extend both the computational and the more linguistically oriented research into language acquisition. We conclude by suggesting how empirical and formal study of language can be best integrated. 1 Unsupervised learning through redundancy reduction Reduction of redundancy is a general (and arguably the only conceivable) approach to unsupervised learning [2, 3]. Written natural language (or transcribed speech) is trivially redundant to the extent it relies on a fixed lexicon. This property of language makes possible the unsupervised recovery of words from a text corpus with all the spaces omitted, through a straightforward minimization of per-letter entropy [4]. Pushing entropy minimization to the limit would lead to an absurd situation in which the agglomeration of words into successively longer “primitive” sequences renders the resulting representation useless for dealing with novel texts (that is, incapable of generalization; cf. [5], p.188). We observe, however, that a word-based representation is still redundant to the extent that different sentences share the same word sequences. Such sequences need not be contiguous; indeed, the detection of paradigmatic variation within a slot in a set of otherwise identical aligned sequences (syntagms) is the basis for the classical distributional theory of language [6], as well as for some modern NLP methods [7]. The pattern — the syntagm and the equivalence class of complementary-distribution symbols that may appear in its open slot — is the main representational building block of our system, ADIOS (for Automatic DIstillation Of Structure) [1].1 Our goal here is to help bridge statistical and formal approaches to language [9] by placing our work on the unsupervised learning of structure in the context of current research in grammar acquisition in computational linguistics, and at the same time to link it to certain formal theories of grammar. Section 2 outlines the main computational principles behind the ADIOS model (for algorithmic details and empirical results, see [1, 10]). Sections 3 and 4 compare our model to select approaches from computational and formal linguistics, respectively. We conclude with a focus on the challenges ahead, discussed in section 5. 2 The principles behind the ADIOS algorithm The representational power of ADIOS and its capacity for unsupervised learning rest on three principles: (1) probabilistic inference of pattern significance, (2) context-sensitive generalization, and (3) recursive construction of complex patterns. Each of these is described briefly below.