Connectionism: past, present, and future

Research efforts to study computation and cognitive modeling on neurally-inspired mechanisms have come to be called Connectionism. Rather than being brand new, it is actually the rebirth of a research programme which thrived from the 40s through the 60s and then was severely retrenched in the 70s. Connectionism is often posed as a paradigmatic competitor to the Symbolic Processing tradition of Artificial Intelligence (Dreyfus & Dreyfus, 1988), and, indeed, the counterpoint in the timing of their intellectual and commercial fortunes may lead one to believe that research in cognition is merely a zero-sum game. This paper surveys the history of the field, often in relation to AI, discusses its current successes and failures, and makes some predictions for where it might lead in the future.

[1]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[2]  James L. McClelland,et al.  Mechanisms of Sentence Processing: Assigning Roles to Constituents of Sentences , 1986 .

[3]  Geoffrey E. Hinton,et al.  Symbols Among the Neurons: Details of a Connectionist Inference Architecture , 1985, IJCAI.

[4]  Bernardo A. Huberman,et al.  AN IMPROVED THREE LAYER, BACK PROPAGATION ALGORITHM , 1987 .

[5]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[6]  Pamela McCorduck,et al.  Machines Who Think: A Personal Inquiry into the History and Prospects of Artificial Intelligence , 1979 .

[7]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[8]  Zenon W. Pylyshyn,et al.  Connectionism and cognitive architecture , 1993 .

[9]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[10]  Geoffrey Sampson,et al.  A Stochastic Approach to Parsing , 1986, COLING.

[11]  Douglas B. Lenat,et al.  The ubiquity of discovery , 1993, AFIPS National Computer Conference.

[12]  A G Barto,et al.  Learning by statistical cooperation of self-interested neuron-like computing elements. , 1985, Human neurobiology.

[13]  Terrence J. Sejnowski,et al.  Analysis of hidden units in a layered network trained to classify sonar targets , 1988, Neural Networks.

[14]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[15]  Garrison W. Cottrell,et al.  A connectionist approach to word sense disambiguation , 1988 .

[16]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[17]  P. Smolensky.  On the proper treatment of connectionism , 1988, Behavioral and Brain Sciences.

[18]  Stephen A. Ritz,et al.  Distinctive features, categorical perception, and probability learning: some applications of a neural model , 1977 .

[19]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: I. An account of basic findings. , 1981 .

[20]  Tad Hogg,et al.  Phase Transitions in Artificial Intelligence Systems , 1987, Artif. Intell..

[21]  D. Robertson,et al.  Studying the earth by very-long-baseline interferometry , 1986 .

[22]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[23]  Teuvo Kohonen,et al.  Storage and Processing of Information in Distributed Associative Memory Systems , 1981 .

[24]  Geoffrey E. Hinton,et al.  Parallel Models of Associative Memory , 1989 .

[25]  Alan H. Kawamoto,et al.  Dynamic processes in the (re)solution of lexical ambiguity , 1985 .

[26]  James L. McClelland,et al.  On learning the past-tenses of English verbs: implicit rules or parallel distributed processing , 1986 .

[27]  James L. McClelland,et al.  An Interactive Activation Model of the Effect of Context in Perception. Part II. Report No. 8003. , 1980 .

[28]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[29]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[30]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[31]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..

[32]  A. Lapedes,et al.  Nonlinear signal processing using neural networks: Prediction and system modelling , 1987 .

[33]  J. Yorke,et al.  Chaos, Strange Attractors, and Fractal Basin Boundaries in Nonlinear Dynamics , 1987, Science.

[34]  Michael Gasser,et al.  Sequencing in a Connectionist Model of Language Processing , 1988, COLING.

[35]  Robert B. Allen,et al.  Several Studies on Natural Language ·and Back-Propagation , 1987 .

[36]  James Gleick.  Chaos: Making a New Science , 1987 .

[37]  W. Ashby,et al.  Design for a brain: The origin of adaptive behaviour (2nd ed. rev.). , 1960 .

[38]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[39]  P. McCorduck.  Machines Who Think , 1979 .

[40]  David E. Rumelhart,et al.  An Interactive Activation Model of the Effect of Context in Perception. Part 2 , 1980 .

[41]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[42]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[43]  Michael I. Jordan.  Serial Order: A Parallel Distributed Processing Approach , 1997 .

[44]  R. Sutton,et al.  Synthesis of nonlinear control surfaces by a layered associative search network , 2004, Biological Cybernetics.

[45]  Teuvo Kohonen,et al.  Associative memory. A system-theoretical approach , 1977 .

[46]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[47]  Lokendra Shastri,et al.  Evidential Reasoning in Semantic Networks: A Formal Theory , 1985, IJCAI.

[48]  C. Sparrow.  The Fractal Geometry of Nature , 1984 .

[49]  M. Minsky.  The Society of Mind , 1986 .

[50]  Bart Selman,et al.  Rule-Based Processing in a Connectionist System for Natural Language Understanding , 1985 .

[51]  Lokendra Shastri,et al.  Semantic Networks: An Evidential Formalization and Its Connectionist Realization , 1988 .

[52]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[53]  Daniel Sabbah,et al.  A connectionist approach to visual recognition , 1982 .

[54]  Mark A. Fanty,et al.  Context-free parsing with connectionist networks , 1987 .

[55]  Donald A. Norman,et al.  Explorations in Cognition , 1975 .

[56]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[57]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: Part 2. The contextual enhancement effect and some tests and extensions of the model. , 1982, Psychological review.

[58]  Hubert L. Dreyfus,et al.  Making a Mind Versus Modelling the Brain , 1990, The Philosophy of Artificial Intelligence.

[59]  Yann LeCun,et al.  Une procedure d'apprentissage pour reseau a seuil asymmetrique (A learning scheme for asymmetric threshold networks) , 1985 .