Connectionist Modeling of Language: Examples and Implications

Researchers interested in human cognitive processes have long used computer simulations to try to identify the principles of cognition. The strategy has been to build computational models that embody putative principles and then to examine how well such models capture human performance in cognitive tasks. Until the early 1980’s, this effort was undertaken largely within the context of the “computer metaphor” of mind. Researchers built computational models based on the conceptualization that the human mind operated as though it were a conventional digital computer. However, with the advent of so-called connectionist, neural network, or parallel distributed processing models (Anderson, Silverstein, Ritz, & Jones, 1977; Hinton & Anderson, 1981; McClelland & Rumelhart, 1981; McClelland, Rumelhart, & PDP Research Group, 1986; Rumelhart, McClelland, & PDP Research Group, 1986), researchers began exploring the implications of principles that are more broadly consistent with the style of computation employed by the brain. In connectionist models, cognitive processes take the form of cooperative and competitive interactions among large numbers of simple, neuron-like processing units (see Figure 1). Unit interactions are governed by weighted connections that collectively encode the long-term knowledge of the system. The activity of some of the units encodes the input to the system; the resulting activity of other units encodes the system’s response to that input. The patterns of activity of the remaining so-called hidden units constitute learned internal representations that mediate between inputs and outputs. Learning involves modifying the values of connection weights based on feedback from the environment on the accuracy of the system’s responses. While each unit exhibits non-linear spatial and temporal summation, units and connections are not generally considered to be in one-to-one correspondence with actual neurons and synapses. Rather, connectionist systems attempt to capture the essential computational properties of the vast ensembles of real neuronal elements found in the brain, through simulations of smaller networks of units. In this way, the approach is distinct from computational neuroscience (Sejnowski, Koch, & Church