Connectionist symbol processing : dead or alive ?

Preface In August 1998 Dave Touretzky asked on the connectionists e-mailing list, \Is connectionist symbol processing dead?" This query lead to an interesting discussion and exchange of ideas. We thought it might be useful to capture this exchange in an article. We solicited contributions, and this collective article is the result. Contributions were solicited by a public call on the connectionists e-mailing list. All contributions received were subjected to two to three informal reviews. Almost all were accepted with varying degrees of revision. Given the number and variety of contributions, the articles cover a wide, though by no means complete, range of the work in the eld. The pieces in this article are of varying nature: position summaries, individual research summaries, historical accounts, discussion of controversial issues, etc. We have not attempted to connect the various pieces together, or to organize them within a coherent framework. Despite this, we think, the reader will nd this collection useful. Implementing symbol processing in networks was a good rst step in solving many problems that plagued symbolic systems. Tony Plate's HRR as applied to analogy is a great example 147]. Using connectionist representations and methodologies, an expensive symbolic similarity estimation process was eliminated in the analogy-making MAC/FAC system 60]. Unfortunately, the entire MAC/FAC hybrid model (like many such models) has a fatal aw that prevents it from leading to an autonomous, exible, creative, intelligent (analogy-making) machine: the overall system organization is still rigidly \symbolic". Their method requires that analogies be encoded as symbols and structures, which leaves no room for perception or context eeects during the analogy making process (for a detailed description of this problem, see Hofstadter 86]). For these reasons, implementing Gentner's rigid framework completely in a neural network (or even real neurons) won't help. Plate's hybrid solution, like most hybrid systems, solved many problems of the MAC/FAC purely-symbolic system. No doubt, hybrid systems are better than their symbolic relatives. However, wherever symbols and structures remain, we seem to be faced with other problems of brittleness and rigidity.

[1]  Jonathan Baxter,et al.  Learning internal representations , 1995, COLT '95.

[2]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[3]  I. Biederman,et al.  Dynamic binding in a neural network for shape recognition. , 1992, Psychological review.

[4]  Frédéric Maire,et al.  Rule-extraction by backpropagation of polyhedra , 1999, Neural Networks.

[5]  Pentti Kanerva,et al.  Binary Spatter-Coding of Ordered K-Tuples , 1996, ICANN.

[6]  Ron Sun,et al.  Robust Reasoning: Integrating Rule-Based and Similarity-Based Reasoning , 1995, Artif. Intell..

[7]  Dan Roth.  A Connectionist Framework for Reasoning: Reasoning with Examples , 1996, AAAI/IAAI, Vol. 2.

[8]  C. Lee Giles,et al.  Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks , 1993 .

[9]  Dan Roth,et al.  Part of Speech Tagging Using a Network of Linear Separators , 1998, ACL.

[10]  Michael J. Healy.  A Topological Semantics for Rule Extraction with Neural Networks , 1999, Connect. Sci..

[11]  Lev Goldfarb,et al.  Why Classical Models for Pattern Recognition are Not Pattern Recognition Models , 1999 .

[12]  C. Lee Giles,et al.  Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution , 1995, IEEE Trans. Neural Networks.

[13]  S. Toulmin.  The uses of argument , 1960 .

[14]  Michael G. Dyer,et al.  High-level Inferencing in a Connectionist Network , 1989 .

[15]  G. Scheler,et al.  Pattern Classiication with Adaptive Distance Measures , 1994 .

[16]  Cora Diamond.  Wittgenstein on Rules and Private Language. , 1983 .

[17]  S. Kraemer.  Steps to an Ecology of Mind: Gregory Bateson , 1993, British Journal of Psychiatry.

[18]  Garrison W. Cottrell,et al.  Time-delay neural networks: representation and induction of finite-state machines , 1997, IEEE Trans. Neural Networks.

[19]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[20]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[21]  N. Pennington,et al.  The story model for juror decision making , 1993 .

[22]  Dominique Perrin,et al.  Finite Automata , 1958, Philosophy.

[23]  T. Sejnowski,et al.  Irresistible environment meets immovable neurons , 1997, Behavioral and Brain Sciences.

[24]  Frédéric Alexandre,et al.  Connectionist-Symbolic Integration: From Unified to Hybrid Approaches , 1996 .

[25]  Tony A. Plate,et al.  Analogy retrieval and processing with distributed vector representations , 2000, Expert Syst. J. Knowl. Eng..

[26]  J Grainger,et al.  Orthographic processing in visual word recognition: a multiple read-out model. , 1996, Psychological review.

[27]  D. Gentner,et al.  Advances in Analogy Research: Integration of Theory and Data from the Cognitive, Computational, and Neural Sciences , 1997, Cognitive Psychology.

[28]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[29]  C. L. Giles,et al.  Dynamic recurrent neural networks: Theory and applications , 1994, IEEE Trans. Neural Networks Learn. Syst..

[30]  Susan I. Hruska,et al.  Back-propagation learning in expert networks , 1992, IEEE Trans. Neural Networks.

[31]  Lokendra Shastri,et al.  ptimally Efficient Inference Syste , 1990 .

[32]  Leslie G. Valiant,et al.  A neuroidal architecture for cognitive computation , 1998, ICALP.

[33]  S C Kleene,et al.  Representation of Events in Nerve Nets and Finite Automata , 1951 .

[34]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[35]  Refractor.  Vision , 2000, The Lancet.

[36]  John S. Bridle,et al.  Alpha-nets: A recurrent 'neural' network architecture with a hidden Markov model interpretation , 1990, Speech Commun..

[37]  Paul Rodríguez,et al.  A Recurrent Neural Network that Learns to Count , 1999, Connect. Sci..

[38]  Paolo Frasconi,et al.  Computational capabilities of local-feedback recurrent networks acting as finite-state machines , 1996, IEEE Trans. Neural Networks.

[39]  Peter W. Foltz,et al.  Learning Human-like Knowledge by Singular Value Decomposition: A Progress Report , 1997, NIPS.

[40]  F. Maire,et al.  A partial order for the M-of-N rule-extraction algorithm , 1997, IEEE Trans. Neural Networks.

[41]  Edmund Leach.  Culture and Communication by Edmund Leach , 1976 .

[42]  G. Edelman,et al.  Real brains and artificial intelligence , 1989 .

[43]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[44]  Vipin Kumar,et al.  Rule-based reasoning in connectionist networks , 1997 .

[45]  Ron Sun,et al.  On Variable Binding in Connectionist Networks , 1992 .

[46]  Jacques Sougné,et al.  Period doubling as a means of representing multiply instantiated entities , 1998 .

[47]  Lokendra Shastri,et al.  Structured connectionist models , 1998 .

[48]  Paul W. B. Atkins,et al.  Models of reading aloud: Dual-route and parallel-distributed-processing approaches. , 1993 .

[49]  Robert M. French,et al.  A Neurobiologically Inspired Model of Working Memory Based on Neuronal Synchrony and Rythmicity , 1997, NCPW.

[50]  Paul Thagard,et al.  Analogical Mapping by Constraint Satisfaction , 1989, Cogn. Sci..

[51]  A. Sperduti.  Labeling Raam , 1994 .

[52]  Marius Usher,et al.  Visual synchrony affects binding and segmentation in perception , 1998, Nature.

[53]  Dennis Norris,et al.  Modeling immediate serial recall with a localist implementation of the primacy model. , 1998 .

[54]  LiMin Fu,et al.  Rule Learning by Searching on Adapted Nets , 1991, AAAI.

[55]  L. Shastri,et al.  From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony , 1993, Behavioral and Brain Sciences.

[56]  A. Jacobs,et al.  Testing a semistochastic variant of the interactive activation model in different word recognition experiments. , 1992, Journal of experimental psychology. Human perception and performance.

[57]  Lokendra Shastri,et al.  Types and Quantifiers in SHRUTI: A Connectionist Model of Rapid Reasoning and Relational Processing , 1998, Hybrid Neural Systems.

[58]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[59]  A. Jacobs,et al.  Models of visual word recognition: Sampling the state of the art. , 1994 .

[60]  Simon M. Lucas,et al.  Syntactic Neural Networks , 1990 .

[61]  Dan Roth,et al.  Learning to reason , 1994, JACM.

[62]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[63]  Simon Mark Lucas.  Connectionist architectures for syntactic pattern recognition , 1991 .

[64]  Lokendra Shastri,et al.  Temporal synchrony, dynamic bindings, and Shruti: A representational but nonclassical model of reflexive reasoning , 1996, Behavioral and Brain Sciences.

[65]  D. R. Mani,et al.  Reflexive Reasoning with Multiple Instantiation in a Connectionist Reasoning System with a Type Hierarchy , 1993 .

[66]  C. von der Malsburg,et al.  Am I Thinking Assemblies , 1986 .

[67]  Stefan Wermter,et al.  Learning dialog act processing , 1996, COLING.

[68]  W. Tabor.  Dynamical Automata , 1998 .

[69]  Stefan Wermter.  Hybrid Connectionist Natural Language Processing , 1994 .

[70]  Petri Kanerva,et al.  Dual role of analogy in the design of a cognitive computer , 1998 .

[71]  W. Singer.  Synchronization of cortical activity and its putative role in information processing and learning. , 1993, Annual review of physiology.

[72]  J. Shavlik,et al.  The Extraction of Reened Rules from Knowledge-based Neural Networks , 1993 .

[73]  Ellen Riloff,et al.  Connectionist, Statistical and Symbolic Approaches to Learning for Natural Language Processing , 1996, Lecture Notes in Computer Science.

[74]  Roger Wales,et al.  Connections, Binding, Unification and Analogical Promiscuity , 1998 .

[75]  John A. Barnden,et al.  Complex Symbol-Processing in Conposit, A Transiently Localist Connectionist Architecture , 1995 .

[76]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[77]  Daniel H. Younger,et al.  Recognition and Parsing of Context-Free Languages in Time n^3 , 1967, Inf. Control..

[78]  Eduardo Sontag,et al.  Turing computability with neural nets , 1991 .

[79]  Gerald M. Edelman,et al.  A Darwinist view of the prospects for conscious artifacts , 1995 .

[80]  Marvin S. Cohen,et al.  Metarecognition in Time-Stressed Decision Making: Recognizing, Critiquing, and Correcting , 1996, Hum. Factors.

[81]  Christoph von der Malsburg,et al.  The Correlation Theory of Brain Function , 1994 .

[82]  Mark S. Cohen.  Three paradigms for viewing decision biases , 1993 .

[83]  Dan Roth,et al.  Learning to Resolve Natural Language Ambiguities: A Unified Approach , 1998, AAAI/IAAI.

[84]  J. Shavlik,et al.  Extracting Reened Rules from Knowledge-based Neural Networks Keywords: Theory Reenement Integrated Learning Representational Shift Rule Extraction from Neural Networks , 1992 .

[85]  F. Saussure,et al.  Course in General Linguistics , 1960 .

[86]  James Henderson,et al.  Description-based parsing in a connectionist network , 1995 .

[87]  W. Singer,et al.  In search of common foundations for cortical computation , 1997, Behavioral and Brain Sciences.

[88]  P. J. Werbos,et al.  Backpropagation: past and future , 1988, IEEE 1988 International Conference on Neural Networks.

[89]  Ron Sun.  The Discrete Neuronal Model and the Probabilistic Discrete Neuronal Model , 1990 .

[90]  Simon M. Lucas,et al.  Forward-Backward Building Blocks for Evolving Neural Networks with Intrinsic Learning Behaviors , 1997, IWANN.

[91]  A. Klopf.  A neuronal model of classical conditioning , 1988 .

[92]  James Henderson,et al.  Connectionist syntactic parsing using temporal variable binding , 1994 .

[93]  B Garner.  A training algorithm for Adaptive Feedforward Neural Networks that determines its own topology , 1998 .

[94]  Brian Hazlehurst,et al.  The Emergence of Propositions from the Co-ordination of Talk and Action in a Shared World , 1998 .

[95]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: I. An account of basic findings. , 1981 .

[96]  D. R. Mani,et al.  The design and implementation of massively parallel knowledge representation and reasoning systems: a connectionist approach , 1996 .

[97]  Ron Sun,et al.  A subsymbolic+symbolic model for learning sequential navigation , 1998 .

[98]  Hava T. Siegelmann,et al.  Computational capabilities of recurrent NARX neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[99]  Stefan Wermter,et al.  SCREEN: learning a flat syntactic and semantic spoken language analysis using artificial neural networks , 1997 .

[100]  John E. Hummel,et al.  Distributed representations of structure: A theory of analogical access and mapping. , 1997 .

[101]  Catherine L. Harris,et al.  The human semantic potential: Spatial language and constrained connectionism , 1997 .

[102]  P. Venkat Rangan,et al.  Adaptive feedback techniques for synchronized multimedia retrieval over integrated networks , 1993, TNET.

[103]  R. Bellman,et al.  Dynamic Programming and Markov Processes , 1960 .

[104]  M. McCloskey.  Networks and Theories: The Place of Connectionism in Cognitive Science , 1991 .

[105]  Jude W. Shavlik,et al.  Using Sampling and Queries to Extract Rules from Trained Neural Networks , 1994, ICML.

[106]  Brian Hazlehurst,et al.  Learning in the Cultural Process , 2000 .

[107]  Yoshua Bengio,et al.  Input-output HMMs for sequence processing , 1996, IEEE Trans. Neural Networks.

[108]  R. Reilly,et al.  Connectionist approaches to natural language processing , 1994 .

[109]  Paul Smolensky,et al.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1990, Artif. Intell..

[110]  Marvin S. Cohen,et al.  Thinking Naturally about Uncertainty , 1996 .

[111]  J E Lisman,et al.  Storage of 7 +/- 2 short-term memories in oscillatory subcycles , 1995, Science.

[112]  Geoffrey E. Hinton,et al.  A Distributed Connectionist Production System , 1988, Cogn. Sci..

[113]  Peter Lane.  Simple synchrony networks : a new connectionist architecture applied to natural language parsing , 1999 .

[114]  S. Grossberg.  Neural Networks and Natural Intelligence , 1988 .

[115]  Michael J. Healy,et al.  Continuous functions and neural network semantics , 1997 .

[116]  M. Pickering,et al.  Architectures and Mechanisms for Language Processing , 1999 .

[117]  Ross W. Gayler,et al.  Multiplicative Binding, Representation Operators & Analogy , 1998 .

[118]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[119]  Alessandro Sperduti,et al.  On the Computational Power of Recurrent Neural Networks for Structures , 1997, Neural Networks.

[120]  Dan Roth,et al.  Reasoning with Models , 1994, Artif. Intell..

[121]  N. Chater,et al.  Computational models and Rethinking innateness , 1999, Journal of Child Language.

[122]  Jacques Sougné,et al.  A Connectionist Model of Reflective Reasoning Using Temporal Properties of Node Firing , 1996 .

[123]  Gerald M. Edelman,et al.  The Remembered Present; A Biological Theory of Consciousness. , 1994 .

[124]  Dan Roth,et al.  Incorporating Knowledge in Natural Language Learning: A Case Study , 1998, WordNet@ACL/COLING.

[125]  S Dehaene,et al.  Neural networks that learn temporal sequences by selection. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[126]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[127]  Francis Heylighen,et al.  Objective, Subjective and Intersubjective Selectors of Knowledge , 1997 .

[128]  Leslie G. Valiant,et al.  Circuits of the mind , 1994 .

[129]  A. Jacobs,et al.  Localist connectionist approaches to human cognition , 1998 .

[130]  G. Bateson,et al.  Mind and Nature: A Necessary Unity , 1979 .

[131]  Jacques Sougné,et al.  Simulation of the Negative Conclusion Bias with INFERNET , 1998 .

[132]  Geoffrey E. Hinton,et al.  Distributed representations and nested compositional structure , 1994 .

[133]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[134]  James Henderson,et al.  Simple Synchrony Networks : Learning to Parse Natural Language with Temporal Synchrony Variable Binding , 1998 .

[135]  Mike Casey,et al.  The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction , 1996, Neural Computation.

[136]  James Henderson,et al.  A Connectionist Architecture for Learning to Parse , 1998, ACL.

[137]  Wolfgang Thomas,et al.  Handbook of Theoretical Computer Science, Volume B: Formal Models and Semantics , 1990 .

[138]  Larry R. Medsker,et al.  Hybrid Intelligent Systems , 1995, Springer US.

[139]  Lawrence A. Bookman.  A Framework for Integrating Relational and Associational Knowledge for Comprehension , 1995 .

[140]  Francis Heylighen,et al.  Selection Criteria for the Evolution of Knowledge , 2000 .

[141]  Marvin S. Cohen,et al.  A Cognitive Framework for Battlefield Commanders' Situation Assessment , 1994 .

[142]  Jonathan Baxter,et al.  The Canonical Distortion Measure for Vector Quantization and Function Approximation , 1997, ICML.

[143]  John P. Miller,et al.  Broadband neural encoding in the cricket cereal sensory system enhanced by stochastic resonance , 1996, Nature.

[144]  Lokendra Shastri,et al.  Rules and Variables in Neural Nets , 1991, Neural Computation.

[145]  Hussein M. Abdel-Wahab,et al.  A Scalable and Robust Feedback Mechanism for Adaptive Multimedia Multicast Systems , 1998, HPN.

[146]  Ron Sun,et al.  Computational Architectures Integrating Neural And Symbolic Processes , 1994 .

[147]  Lokendra Shastri,et al.  A Connectionist Encoding of Schemas and Reactive Plans , 1997 .

[148]  Pekka Orponen,et al.  On the Effect of Analog Noise in Discrete-Time Analog Computations , 1996, Neural Computation.

[149]  B Garner.  A symbolic solution for adaptive feedforward neural networks found with a new training algorithm , 1998 .

[150]  Alessandro Sperduti,et al.  A general framework for adaptive processing of data structures , 1998, IEEE Trans. Neural Networks.

[151]  Sara Shute.  Understanding Language Acquisition: The Framework of Learning , 1997 .

[152]  Jared Freeman,et al.  Critical thinking skills in tactical decision making: A model and a training strategy. , 1998 .

[153]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[154]  E. Vogel,et al.  Response from Luck and Vogel , 1998, Trends in Cognitive Sciences.

[155]  Claude Lévi-Strauss,et al.  Structural Anthropology Volume II , 1977 .

[156]  Jared Freeman,et al.  Time-Stressed Decision-Making in the Cockpit , 1998 .

[157]  Tony Plate,et al.  Holographic Reduced Representations: Convolution Algebra for Compositional Distributed Representations , 1991, IJCAI.

[158]  Ron Sun,et al.  A Bottom-Up Model of Skill Learning , 1998 .

[159]  Garrison W. Cottrell,et al.  Grounding Meaning in Perception , 1990, GWAI.

[160]  Jacques Sougné,et al.  Connectionism and the problem of multiple instantiation , 1998, Trends in Cognitive Sciences.

[161]  Melanie Mitchell,et al.  Analogy-making as perception - a computer model , 1993, Neural network modeling and connectionism.

[162]  Jerome A. Feldman,et al.  Neural Representation of Conceptual Knowledge. , 1986 .

[163]  L. Shastri,et al.  A Connectionist Treatment of Negation and Inconsistency , 1996 .

[164]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[165]  Lisa Meeden.  Towards planning: incremental investigations into adaptive robot control , 1995 .

[166]  S. Harnad.  Connecting Object to Symbol in Modeling Cognition , 1992 .

[167]  Stefan C. Kremer,et al.  Finite State Automata that Recurrent Cascade-Correlation Cannot Represent , 1995, NIPS.

[168]  John A. Barnden,et al.  Overcoming Rule-Based Rigidity and Connectionist Limitations through Massively-Parallel Case-Based Reasoning , 1992, Int. J. Man Mach. Stud..

[169]  D. Levine.  Introduction to Neural and Cognitive Modeling , 2018 .

[170]  Bryan Thompson,et al.  Metacognitive Behavior in Adaptive Agents , 1998 .

[171]  C. Lee Giles,et al.  Constructing deterministic finite-state automata in recurrent neural networks , 1996, JACM.

[172]  Stefan Wermter,et al.  Building lexical representations dynamically using arti cial neural networks , 1997 .

[173]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[174]  Cristopher Moore,et al.  Dynamical Recognizers: Real-Time Language Recognition by Analog Computers , 1998, Theor. Comput. Sci..

[175]  Hava T. Siegelmann,et al.  The Simple Dynamics of Super Turing Theories , 1996, Theor. Comput. Sci..

[176]  J. Baker.  Trainable grammars for speech recognition , 1979 .