Learning Distributed Representations for Syllables

This paper presents a connectionist model of how representations for syllables might be learned from sequences of phones. A simple recurrent network is trained to distinguish a set of words in an artificial language, which are presented to it as sequences of phonetic feature vectors. The distributed syllable representations that are learned as a side-effect of this task are used as input to other networks. It is shown that these representations encode syllable structure in a way which permits the regeneration of the phone sequences (for production) as well as systematic phonological operations on the representations. Linguistic Structure and Distributed Representation If the language sciences agree on one thing, it is the hierarchical nature of language. The importance of hierarchical, structured representations is now generally recognized for the phonological pole, where syllables and metrical units now play a major role (see, e.g., Frazier (1987) and Goldsmith (1990)), as well as for the syntactic/semantic pole of language and language processing. The major reason for believing in structured representations is the significance of structure-sensitive operations in language processing. A semantic inference rule may need to know where the subject of a clause is; a morphological reduplication rule may need to know where the coda (final consonant(s)) of a syllable is. Traditional symbolic representations are based crucially on the simple notion of concatenation (van Gelder, 1990). A syllable representation, for example, is a (bracketed) string of concatenated phones. Recent connectionist work offers as an alternative to this widely accepted approach distributed representations, for which it is generally impossible to isolate which elements of the representation denote which of the lower-level units comprising the structure being represented. What good are distributed representations? They certainly are harder to interpret directly, at least by external “users” of the system that creates them. And In Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society (1992), 396–401 at first blush it seems cumbersome, if not impossible, to implement structure-sensitive operations on them, operations which present no particular difficulty for symbolic representations (Fodor & Pylyshyn, 1988). Clearly distributed representations would be useless for most purposes if they were not amenable to such operations. Recently, however, it has been shown that it is possible to arrive at a set of connection weights which implements structure-sensitive operations on distributed representations. Where the representations arise on hidden layers through training, the operations on them are also implemented through training (Chalmers, 1990). Where the representations arise as a result of the application of a set of primitive operations analogous to the filling of roles in symbolic models, the operations on them can be implemented more directly (Legendre, Miyata, & Smolensky, 1991). There are three reasons to prefer distributed over symbolic representations for structured objects such as syllables and sentences. 1. Distributed representations do not necessarily increase in size as the complexity of the represented object increases. In the case of some types of representations, for example, those described in this paper, representations for objects of the same type are of fixed width (Pollack, 1990). This seems more important for syntax/semantics than for phonology, where there is apparently no recursive embedding, but in a learning context, it is a desirable feature for phonological representations too since a system cannot be expected to know beforehand how complex the representations will need to be and therefore how much memory to allot to them. 2. Complex transformations can be performed on distributed representations in a single parallel step, rather than through a series of symbolic conses, cars, and cdrs (Legendre et al., 1991). 3. There are relatively simple algorithms for learning the structure in distributed representations (Elman, 1990; Pollack, 1990). Most work concerned with distributed representations for structured objects has examined syntax or semantics. It remains to be shown whether it is possible to learn distributed syllable representations which embody the structure required for phonological operations of various sorts. This is in part what this study seeks to establish.