Neural networks for word recognition: Is a hidden layer necessary?

Neural networks for word recognition: Is a hidden layer necessary? Frederic Dandurand (Frederic.Dandurand@univ-provence.fr) Laboratoire de Psychologie Cognitive, CNRS, Aix-Marseille University 3, place Victor Hugo, 13331 Marseille, France Thomas Hannagan (thom.hannagan@gmail.com) Laboratoire de Sciences Cognitives et Psycholinguistique, EHESS/CNRS/DEC-ENS, Ecole Normale Superieure 29 rue d’Ulm, 75005 Paris Jonathan Grainger (jonathan.grainger@univ-provence.fr) Laboratoire de Psychologie Cognitive, CNRS, Aix-Marseille University 3, place Victor Hugo, 13331 Marseille, France primes formed of a subset of the target’s letters (e.g., grdn- garden) compared with a prime formed of the same subset of letters in the wrong order (e.g., gdrn-garden). A number of models have been proposed for an intermediate level of coding that can account for these priming effects (see Grainger, 2008 for a review). Notably, the Grainger and Van Heuven (2003) model of orthographic processing was the inspiration for a computational model that learned to map location-specific letter identities (letters coded as a function of their position in a horizontal array) onto location-invariant lexical representations (Dandurand, Grainger, & Dufau, 2010). Because parsimony dictates to assume a single intermediate level of representation, we considered a neural network architecture with a single hidden layer. This network architecture with a hidden layer successfully captured transposed-letter and relative-position priming effects (Dandurand et al., 2010). Intermediate representations were explicitly probed and analyzed as patterns of activation at the hidden layer (Hannagan, Dandurand, & Grainger, submitted; see also Plaut, McClelland, Seidenberg, & Patterson 1996 for a discussion of internal representations in neural networks). These patterns were found to have two important characteristics. First, letters seemed to be represented in a semi-location- invariant fashion at the hidden layer. Second, representations at the hidden layer were well-characterized as a holographic overlap coding in which small changes of the inputs resulted in small differences in hidden layer representations. More specifically, differences in patterns of hidden layer activations were monotonically related to differences in identity and position of input letters. For example, patterns of activity at the hidden layer were more different for a two-letter substitution at the input (POLL vs. BULL) than a single letter substitution (PULL vs. BULL) when position in the horizontal array was kept constant. Furthermore, differences in patterns of activity were also larger when the input word was moved by two positions in the alphabetic array (#THAT##### vs. ###THAT###) than moved by a single position (#THAT##### vs. ##THAT####). Holographic overlap coding explains the observed transposed-letter and relative-position priming and Abstract We study neural network models that learn location invariant orthographic representations for printed words. We compare two model architectures: with and without a hidden layer. We find that both architectures succeed in learning the training data and in capturing benchmark phenomena of skilled reading – transposed-letter and relative-position priming. Networks without a hidden layer use a strategy for identifying target words based on the presence of letters in the target word, but where letter contributions are modulated using the interaction between within-word position and within-slot location. This modulation allows networks to factor in some information about letter position, which is sufficient to segregate most anagrams. The hidden layer appears critical for success in a lexical decision task, i.e., sorting words from non-words. Networks with a hidden layer better succeed at correctly rejecting non-words than networks without a hidden layer. The latter tend to over-generalize and confuse non- words for words that share letters. Keywords: Computational modeling, word recognition, neural networks, reading, priming effects. Introduction An important cognitive activity involved in skilled reading is the mapping of retinal images of letters onto abstract word representations. Skilled readers can identify words relatively easily (although not perfectly, see e.g., Rayner, White, Johnson, Liversedge, 2006) even when letter order is jumbled, except for the first and last letters. This suggests that at least one intermediate level of coding exists that abstracts away from absolute letter position and instead codes some information about relative letter order. Such an intermediate level of representation has been studied using a number of techniques including masked priming (see Grainger, 2008 for a review). Robust priming effects found include the transposed-letter effect and the relative-position effect. The transposed-letter effect describes the superior priming observed from primes formed by transposing two of the target’s letters (e.g., gadren-garden) compared with primes formed by substituting two of the target’s letters (e.g., galsen-garden). The relative-position priming effect describes a processing advantage for targets preceded by