Birth of an Abstraction: A Dynamical Systems Account of the Discovery of an Elsewhere Principle in a Category Learning Task

Human participants and recurrent ("connectionist") neural networks were both trained on a categorization system abstractly similar to natural language systems involving irregular ("strong") classes and a default class. Both the humans and the networks exhibited staged learning and a generalization pattern reminiscent of the Elsewhere Condition (Kiparsky, 1973). Previous connectionist accounts of related phenomena have often been vague about the nature of the networks' encoding systems. We analyzed our network using dynamical systems theory, revealing topological and geometric properties that can be directly compared with the mechanisms of non-connectionist, rule-based accounts. The results reveal that the networks "contain" structures related to mechanisms posited by rule-based models, partly vindicating the insights of these models. On the other hand, they support the one mechanism (OM), as opposed to the more than one mechanism (MOM), view of symbolic abstraction by showing how the appearance of MOM behavior can arise emergently from one underlying set of principles. The key new contribution of this study is to show that dynamical systems theory can allow us to explicitly characterize the relationship between the two perspectives in implemented models.

[1]  Carol Lynn Moder,et al.  Morphological Classes as Natural Categories , 1983 .

[2]  S. Pinker,et al.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition , 1988, Cognition.

[3]  Michael K. Tanenhaus,et al.  Dynamical models of sentence processing , 1999, Cogn. Sci..

[4]  Rita M. Manzini,et al.  Parameters, binding theory and learnability , 1987 .

[5]  Béatrice Duval,et al.  Representation of Incomplete Knowledge by Induction of Default Theories , 2001, LPNMR.

[6]  Terry Regier,et al.  The Human Semantic Potential: Spatial Language and Constrained Connectionism , 1996 .

[7]  Joan L. Bybee,et al.  Rules and schemas in the development and use of the English past tense , 1982 .

[8]  James L. McClelland,et al.  Letting structure emerge: connectionist and dynamical systems approaches to cognition , 2010, Trends in Cognitive Sciences.

[9]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[10]  R. Shepard,et al.  Toward a universal law of generalization for psychological science. , 1987, Science.

[11]  Karl J. Friston.  Learning and inference in the brain , 2003, Neural Networks.

[12]  Robert M. French,et al.  Noise and the Emergence of Rules in Category Learning: A Connectionist Model , 2011, IEEE Transactions on Autonomous Mental Development.

[13]  Paco Calvo,et al.  How Many Mechanisms Are Needed to Analyze Speech? A Connectionist Simulation of Structural Rule Learning in Artificial Language Acquisition , 2011, Cogn. Sci..

[14]  James L. McClelland,et al.  On learning the past-tenses of English verbs: implicit rules or parallel distributed processing , 1986 .

[15]  T. Jaeger,et al.  Categorical Data Analysis: Away from ANOVAs (transformation or not) and towards Logit Mixed Models. , 2008, Journal of memory and language.

[16]  B. Hayes,et al.  Rules vs. analogy in English past tenses: a computational/experimental study , 2003, Cognition.

[17]  Scott Hotton,et al.  Extending Dynamical Systems Theory to Model Embodied Cognition , 2011, Cogn. Sci..

[18]  James L. McClelland,et al.  Rules or connections in past-tense inflections: what does the evidence rule out? , 2002, Trends in Cognitive Sciences.

[19]  Gregory Ashby,et al.  A neuropsychological theory of multiple systems in category learning. , 1998, Psychological review.

[20]  Martina Penke,et al.  The Representation of Inflectional Morphology: Evidence from Broca's Aphasia , 1999, Brain and Language.

[21]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[22]  George Sperling,et al.  The information available in brief visual presentations. , 1960 .

[23]  M Coltheart,et al.  DRC: a dual route cascaded model of visual word recognition and reading aloud. , 2001, Psychological review.

[24]  Peter Hagoort,et al.  A PET study of cerebral activation patterns induced by verb inflection , 1997 .

[25]  J. Kruschke.  Bayesian approaches to associative learning: From passive to active learning , 2008, Learning & behavior.

[26]  J. Jaeger.  A POSITRON EMISSION TOMOGRAPHIC STUDY OF REGULAR AND IRREGULAR VERB MORPHOLOGY IN ENGLISH , 1996 .

[27]  Gary M. Oppenheim,et al.  The dark side of incremental learning: A model of cumulative semantic interference during lexical access in speech production , 2010, Cognition.

[28]  Michael J Cortese,et al.  Consistency and regularity in past-tense verb generation in healthy ageing, Alzheimer's disease, and semantic dementia , 2006, Cognitive neuropsychology.

[29]  R. French,et al.  A connectionist account of asymmetric category learning in early infancy. , 2000, Developmental psychology.

[30]  Paul D. Bartos,et al.  Connectionist modelling of category learning , 2002 .

[31]  Fernando J. Pineda.  Recurrent backpropagation networks , 1995 .

[32]  S. Pinker,et al.  A Neural Dissociation within Language: Evidence that the Mental Dictionary Is Part of Declarative Memory, and that Grammatical Rules Are Processed by the Procedural System , 1997, Journal of Cognitive Neuroscience.

[33]  C. Cazden.  The acquisition of noun and verb inflections. , 1968, Child development.

[34]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[35]  Harald Clahsen,et al.  Syntax and morphology in Williams syndrome , 1998, Cognition.

[36]  Whitney Tabor,et al.  A dynamical systems perspective on the relationship between symbolic and non-symbolic computation , 2009, Cognitive Neurodynamics.

[37]  Marina Nespor,et al.  Signal-Driven Computations in Speech Processing , 2002, Science.

[38]  R. Nosofsky.  Attention, similarity, and the identification-categorization relationship. , 1986, Journal of experimental psychology. General.

[39]  B MacWhinney,et al.  Frequency and the lexical storage of regularly inflected forms , 1986, Memory & cognition.

[40]  J. Shonkoff,et al.  Development of infants with disabilities and their families: implications for theory and service delivery. , 1992, Monographs of the Society for Research in Child Development.

[41]  M. Maratsos.  More overregularizations after all: new data and discussion on Marcus, Pinker, Ullman, Hollander, Rosen & Xu , 2000, Journal of Child Language.

[42]  Michael K. Tanenhaus,et al.  Parsing in a Dynamical System: An Attractor-based Account of the Interaction of Lexical and Structural Constraints in Sentence Processing , 1997 .

[43]  A. Endress,et al.  Rapid learning of syllable classes from a perceptually continuous speech stream , 2007, Cognition.

[44]  Chris Collins,et al.  The handbook of contemporary syntactic theory , 2001 .

[45]  Raymond Reiter,et al.  A Logic for Default Reasoning , 1987, Artif. Intell..

[46]  Barak A. Pearlmutter.  Gradient calculations for dynamic recurrent neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[47]  Janet Dean Fodor.  Setting Syntactic Parameters , 2008 .

[48]  Lothar M. Schmitt,et al.  An ER-fMRI investigation of morphological inflection in German reveals that the brain makes a distinction between regular and irregular forms , 2003, Brain and Language.

[49]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[50]  S. Pinker,et al.  The past and future of the past tense , 2002, Trends in Cognitive Sciences.

[51]  S Pinker,et al.  Sensitivity of children's inflection to grammatical structure , 1994, Journal of Child Language.

[52]  Kim Plunkett,et al.  A Connectionist Model of the Arabic Plural System , 1997 .

[53]  Jeffrey L. Elman,et al.  Default Generalisation in Connectionist Networks. , 1995 .

[54]  Anuenue Kukona.  Self-organization in anticipatory language contexts: A new view of top-down and bottom-up constraint integration during online sentence processing , 2011 .

[55]  Steven Pinker,et al.  Generalizations of regular and irregular morphology , 1993 .

[56]  Whitney Tabor,et al.  Syntactic innovation : a connectionist model , 1994 .

[57]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[58]  John F. Kolen,et al.  Gradient Calculations for Dynamic Recurrent Neural Networks , 2001 .

[59]  Steven Pinker,et al.  Why No Mere Mortal Has Ever Flown Out to Center Field , 1991, Cogn. Sci..

[60]  Joan L. Bybee.  Morphology: A study of the relation between meaning and form , 1985 .

[61]  Patrick Juola,et al.  A connectionist model of english past tense and plural morphology , 1999, Cogn. Sci..

[62]  S. Pinker,et al.  Combination and structure, not gradedness, is the issue , 2002, Trends in Cognitive Sciences.

[63]  S Pinker,et al.  Overregularization in language acquisition. , 1992, Monographs of the Society for Research in Child Development.

[64]  John P. McDermott,et al.  OPS, A Domain-Independent Production System Language , 1977, IJCAI.

[65]  J. Tenenbaum,et al.  Word learning as Bayesian inference. , 2007, Psychological review.

[66]  Michael T. Ullman,et al.  Inflectional morphology in a family with inherited specific language impairment , 1999, Applied Psycholinguistics.

[67]  William D. Marslen-Wilson,et al.  Dissociating types of mental computation , 1997, Nature.

[68]  J. Kruschke,et al.  ALCOVE: an exemplar-based connectionist model of category learning. , 1992, Psychological review.

[69]  J. Berko.  The Child's Learning of English Morphology , 1958 .

[70]  S Pinker,et al.  Weird past tense forms , 1995, Journal of Child Language.