Connectionist inference models

The performance of symbolic inference tasks has long been a challenge to connectionists. In this paper, we present an extended survey of this area. Existing connectionist inference systems are reviewed, with particular reference to how they perform variable binding and rule-based reasoning and whether they involve distributed or localist representations. The benefits and disadvantages of different representations and systems are outlined, and conclusions drawn regarding the capabilities of connectionist inference systems when compared with symbolic inference systems or when used for cognitive modelling.

[1]  P. Picton,et al.  TWO ANALYSIS TECHNIQUES FOR FEED-FORWARD NETWORKS , 1999 .

[2]  Antony Browne.  Performing a symbolic inference step on distributed representations , 1998, Neurocomputing.

[3]  Nikola K. Kasabov.  Connectionist Fuzzy Production Systems , 1993, Fuzzy Logic in Artificial Intelligence.

[4]  W. Freeman.  Second Commentary: On the proper treatment of connectionism by Paul Smolensky (1988) - Neuromachismo Rekindled , 1989 .

[5]  J. A. Robinson,et al.  A Machine-Oriented Logic Based on the Resolution Principle , 1965, JACM.

[6]  Vipin Kumar,et al.  Rule-based reasoning in connectionist networks , 1997 .

[7]  Abraham Kandel,et al.  Hybrid Architectures for Intelligent Systems , 1992 .

[8]  Ron Sun,et al.  On Variable Binding in Connectionist Networks , 1992 .

[9]  Allan Collins,et al.  A spreading-activation theory of semantic processing , 1975 .

[10]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[11]  Venkat Ajjanagadde,et al.  Incorporating background knowledge and structured explananda in abductive reasoning: a framework , 1993, IEEE Trans. Syst. Man Cybern..

[12]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[13]  Melanie Hilario.  Architectures and techniques for knowledge-based neurocomputing , 2000 .

[14]  Stefan Wermter,et al.  Hybrid neural systems: from simple coupling to fully integrated neural networks , 1999 .

[15]  Jude W. Shavlik,et al.  Understanding Time-Series Networks: A Case Study in Rule Extraction , 1997, Int. J. Neural Syst..

[16]  Alfred Ultsch,et al.  A connectionist knowledge-acquisition tool: CONKAT , 1993 .

[17]  S. Phillips,et al.  Processing capacity defined by relational complexity: implications for comparative, developmental, and cognitive psychology. , 1998, The Behavioral and brain sciences.

[18]  Michael G. Dyer,et al.  Learning Distributed Representations of Conceptual Knowledge and their Application to Script-based Story Processing , 1990 .

[19]  Armando Freitas da Rocha,et al.  Inference, inquiry, evidence censorship, and explanation in connectionist expert systems , 1997, IEEE Trans. Fuzzy Syst..

[20]  Liya Ding,et al.  A Prolog-like inference system based on neural logic - An attempt towards fuzzy neural logic programming , 1996, Fuzzy Sets Syst..

[21]  Guido Bologna,et al.  Lessons from Past, Current Issues, and Future Research Directions in Extracting the Knowledge Embedded in Artificial Neural Networks , 1998, Hybrid Neural Systems.

[22]  Noel E. Sharkey.  The ghost in the hybrid-a study of uniquely connectionist representations , 1992 .

[23]  John A. Bullinaria,et al.  Analyzing the Internal Representations of Trained Neural Networks , 1997 .

[24]  L. Shastri,et al.  Knowledge Fusion in the Large - taking a cue from the brain , 1999 .

[25]  Marvin S. Cohen,et al.  Metarecognition in Time-Stressed Decision Making: Recognizing, Critiquing, and Correcting , 1996, Hum. Factors.

[26]  Geoffrey E. Hinton,et al.  Learning Distributed Representations of Concepts Using Linear Relational Embedding , 2001, IEEE Trans. Knowl. Data Eng..

[27]  Melanie Hilario,et al.  An Overview of Strategies for Neurosymbolic Integration , 1995 .

[28]  Dominic Palmer-Brown,et al.  (S)RAAM: An Analytical Technique for Fast and Reliable Derivation of Connectionist Symbol Structure Representations , 1997, Connect. Sci..

[29]  Antony Browne,et al.  Connectionist variable binding , 1999, Expert Syst. J. Knowl. Eng..

[30]  Jude Shavlik,et al.  An Approach to Combining Explanation-based and Neural Learning Algorithms , 1989 .

[31]  Geoffrey E. Hinton,et al.  Mundane Reasoning by Parallel Constraint Satisfaction , 1990 .

[32]  Waldo C. Kabat,et al.  Automated Synthesis of Combinational Logic Using Theorem-Proving Techniques , 1985, IEEE Transactions on Computers.

[33]  Zdravko Markov.  A Tool for Building Connectionist-like Networks Based on Term Unification , 1991, PDK.

[34]  Geoffrey E. Hinton.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[35]  Ignacio Requena,et al.  Are artificial neural networks black boxes? , 1997, IEEE Trans. Neural Networks.

[36]  Michael G. Dyer,et al.  Distributed symbol formation and processing in connectionist networks , 1990, J. Exp. Theor. Artif. Intell..

[37]  Paul Buchheit.  A neuro‐propositional model of language processing , 1999 .

[38]  R. Nakano,et al.  Medical diagnostic expert system based on PDP model , 1988, IEEE 1988 International Conference on Neural Networks.

[39]  Lars Niklasson,et al.  Can Connectionist Models Exhibit Non-Classical Structure Sensitivity? , 2019, Proceedings of the Sixteenth Annual Conference of the Cognitive Science Society.

[40]  Gadi Pinkas,et al.  Reasoning, Nonmonotonicity and Learning in Connectionist Networks that Capture Propositional Knowledge , 1995, Artif. Intell..

[41]  Lokendra Shastri,et al.  Rules and Variables in Neural Nets , 1991, Neural Computation.

[42]  Ron Sun,et al.  Computational Architectures Integrating Neural And Symbolic Processes , 1994 .

[43]  Lokendra Shastri,et al.  A Connectionist Encoding of Schemas and Reactive Plans , 1997 .

[44]  Jared Freeman,et al.  Metarecognition in Time-Stressed Decision Making: Recognizing, Critiquing, and Correcting , 1996, Hum. Factors.

[45]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..

[46]  LiMin Fu,et al.  Rule Generation from Neural Networks , 1994, IEEE Trans. Syst. Man Cybern. Syst..

[47]  L. Wos,et al.  Paramodulation and Theorem-Proving in First-Order Theories with Equality , 1983 .

[48]  Stefan Wermter,et al.  Knowledge extraction from radial basis function networks and multilayer perceptrons , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[49]  P. Smolensky.  On the proper treatment of connectionism , 1988, Behavioral and Brain Sciences.

[50]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[51]  J. A. Robinson,et al.  Automatic Deduction with Hyper-Resolution , 1983 .

[52]  T. Gelder,et al.  Mind as Motion: Explorations in the Dynamics of Cognition , 1995 .

[53]  Chris Eliasmith,et al.  Integrating structure and meaning: a distributed model of analogical mapping , 2001, Cogn. Sci..

[54]  R. C. Lacher,et al.  Extracting rules by destructive learning , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[55]  Samuel W. K. Chan,et al.  Symbolic connectionism in natural language disambiguation , 1998, IEEE Trans. Neural Networks.

[56]  Robert Wadley,et al.  Connectionism , Rule Following , and Symbolic Manipulatio , 1999 .

[57]  Ross W. Gayler,et al.  Multiplicative Binding, Representation Operators & Analogy , 1998 .

[58]  Omid M. Omidvar.  Progress in neural networks , 1991 .

[59]  Michael C. Mozer,et al.  Dynamic Conflict Resolution in a Connectionist Rule-Based System , 1993, International Joint Conference on Artificial Intelligence.

[60]  Antony Browne,et al.  Neural Network Perspectives on Cognition and Adaptive Robotics , 1997 .

[61]  Masumi Ishikawa,et al.  Rule extraction by successive regularization , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[62]  Christopher John Hogger,et al.  Essentials of logic programming , 1990 .

[63]  Dmitri A. Rachkovskij,et al.  Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning , 2001, Neural Computation.

[64]  Stephen I. Gallant,et al.  Connectionist expert systems , 1988, CACM.

[65]  J. M. Ben,et al.  Are Arti cial Neural Networks Black Boxes ? , 1996 .

[66]  Rudy Setiono.  Extracting M-of-N rules from trained neural networks , 2000, IEEE Trans. Neural Networks Learn. Syst..

[67]  M. Page,et al.  Connectionist modelling in psychology: A localist manifesto , 2000, Behavioral and Brain Sciences.

[68]  Jack Perkins,et al.  Pattern recognition in practice , 1980 .

[69]  Ron Sun,et al.  A new approach toward modeling causality in commonsense reasoning , 1995, Int. J. Intell. Syst..

[70]  Lawrence J. Henschen,et al.  Unit Refutations and Horn Sets , 1974, JACM.

[71]  Stefan Wermter,et al.  A Novel Modular Neural Architecture for Rule-Based and Similarity-Based Reasoning , 1998, Hybrid Neural Systems.

[72]  Bruce J. MacLennan,et al.  Characteristics of connectionist knowledge representation , 1991, Inf. Sci..

[73]  Ron Sun,et al.  Autonomous learning of sequential tasks: experiments and analyses , 1998, IEEE Trans. Neural Networks.

[74]  Graeme S. Halford,et al.  Systematicity: Psychological evidence with connectionist implications , 1997 .

[75]  G. P. Fletcher,et al.  Using neural networks as a tool for constructing rule based systems , 1995, Knowl. Based Syst..

[76]  Guido Bologna,et al.  Rule extraction from a multilayer perceptron with staircase activation functions , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[77]  Géraldine Legendre,et al.  Principles for an Integrated Connectionist/Symbolic Theory of Higher Cognition ; CU-CS-600-92 , 1992 .

[78]  Ron Sun,et al.  Robust Reasoning: Integrating Rule-Based and Similarity-Based Reasoning , 1995, Artif. Intell..

[79]  Nikola Kasabov,et al.  A Connectionist Production System with Partial Match and its Use for Approximate Reasoning , 1993 .

[80]  Francis Crick,et al.  The recent excitement about neural networks , 1989, Nature.

[81]  Terrence J. Sejnowski,et al.  The Computational Brain , 1996, Artif. Intell..

[82]  Sushmita Mitra,et al.  Neuro-fuzzy rule generation: survey in soft computing framework , 2000, IEEE Trans. Neural Networks Learn. Syst..

[83]  Peter M. Vishton,et al.  Rule learning by seven-month-old infants. , 1999, Science.

[84]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[85]  Andrew Heathcote,et al.  The Law of Practice and localist neural network models , 2000, Behavioral and Brain Sciences.

[86]  Frédéric Alexandre,et al.  Connectionist-Symbolic Integration: From Unified to Hybrid Approaches , 1996 .

[87]  Tony A. Plate,et al.  Analogy retrieval and processing with distributed vector representations , 2000, Expert Syst. J. Knowl. Eng..

[88]  Michael G. Dyer,et al.  Connectionist Natural Language Processing: A Status Report , 1995 .

[89]  Antônio de Pádua Braga,et al.  Extracting Rules from Neural Networks: a , 1998 .

[90]  Alessandro Sperduti,et al.  Stability properties of labeling recursive auto-associative memory , 1995, IEEE Trans. Neural Networks.

[91]  Todd Peterson,et al.  An RBF network alternative for a hybrid architecture , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[92]  C. L. Giles,et al.  Rule refinement with recurrent neural networks , 1993, IEEE International Conference on Neural Networks.

[93]  Joachim Diederich,et al.  Connectionist Recruitment Learning , 1988, ECAI.

[94]  Jude Shavlik,et al.  THE EXTRACTION OF REFINED RULES FROM KNOWLEDGE BASED NEURAL NETWORKS , 1993 .

[95]  Allen Newell,et al.  Physical Symbol Systems , 1980, Cogn. Sci..

[96]  L. Magdalena.  A rst approach to a Taxonomy of Fuzzy Neural Systems � , 1995 .

[97]  John A. Barnden.  Neural-Net Implementation of Complex Symbol-Processing in a Mental Model Approach to Syllogistic Reasoning , 1989, IJCAI.

[98]  Peter Géczy,et al.  Rule Extraction from Trained Artificial Neural Networks , 1997, ICONIP.

[99]  Yoichi Hayashi,et al.  A Neural Expert System with Automated Extraction of Fuzzy If-Then Rules , 1990, NIPS.

[100]  Stuart C. Shapiro,et al.  Encyclopedia of artificial intelligence, vols. 1 and 2 (2nd ed.) , 1992 .

[101]  Kenji Baba,et al.  Explicit representation of knowledge acquired from plant historical data using neural network , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[102]  Joydeep Ghosh,et al.  Evaluation and ordering of rules extracted from feedforward networks , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[103]  Larry R. Medsker,et al.  Hybrid Neural Network and Expert Systems , 1994, Springer US.

[104]  J. Shavlik,et al.  The Extraction of Reened Rules from Knowledge-based Neural Networks , 1993 .

[105]  Dana H. Ballard,et al.  Parallel Logical Inference and Energy Minimization , 1986, AAAI.

[106]  Ronald Rosenfeld,et al.  Coarse-Coded Symbol Memories and Their Properties , 1988, Complex Syst..

[107]  L. Shastri,et al.  From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony , 1993, Behavioral and Brain Sciences.

[108]  Jerome A. Feldman,et al.  Extending Embodied Lexical Development , 1998 .

[109]  A. Browne.  Detecting systematic structure in distributed representations , 1998, Neural Networks.

[110]  Geoffrey E. Hinton.  Mapping Part-Whole Hierarchies into Connectionist Networks , 1990, Artif. Intell..

[111]  Charles P. Dolan,et al.  Tensor Product Production System: a Modular Architecture and Representation , 1989 .

[112]  Stefan Wermter,et al.  Tensor models: A creative basis for memory retrieval and analogical mapping , 1992 .

[113]  Martin J. Adamson,et al.  B-RAAM: A Connectionist Model which Develops Holistic Internal Representations of Symbolic Structures , 1999, Connect. Sci..

[114]  Nils J. Nilsson,et al.  Problem-solving methods in artificial intelligence , 1971, McGraw-Hill computer science series.

[115]  Anca L. Ralescu,et al.  A connectionist approach for rule-based inference using an improved relaxation method , 1992, IEEE Trans. Neural Networks.

[116]  James Henderson,et al.  Simple Synchrony Networks : Learning to Parse Natural Language with Temporal Synchrony Variable Binding , 1998 .

[117]  Steven Phillips,et al.  Are Feedforward and Recurrent Networks Systematic? Analysis and Implications for a Connectionist Cognitive Architecture , 1998, Connect. Sci..

[118]  S. Pinker,et al.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition , 1988, Cognition.

[119]  Yoichi Hayashi,et al.  A Neural Network Expert System with Confidence Measurements , 1990, IPMU.

[120]  Pentti Kanerva,et al.  Encoding Structure in Boolean Space , 1998 .

[121]  C. Lee Giles,et al.  Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks , 1993 .

[122]  Volker Weber.  Uniication in Prolog by Connectionist Models , 1993 .

[123]  Michael G. Dyer,et al.  High-level Inferencing in a Connectionist Network , 1989 .

[124]  S. Kumar,et al.  Bipolar radial basis function inferencing networks , 1997, Neurocomputing.

[125]  Sebastian Thrun,et al.  Extracting Rules from Artifical Neural Networks with Distributed Representations , 1994, NIPS.

[126]  D. J. Hand,et al.  Artificial Intelligence Frontiers in Statistics: AI and Statistics III , 1992 .

[127]  John A. Barnden,et al.  Overcoming Rule-Based Rigidity and Connectionist Limitations through Massively-Parallel Case-Based Reasoning , 1992, Int. J. Man Mach. Stud..

[128]  Daniel S. Yeung,et al.  Knowledge Matrix - An Explanation & Knowledge Refinement Facility for a Rule Induced Neural Network , 1994, AAAI.

[129]  Tiong-Hwee Goh.  Semantic extraction using neural network modelling and sensitivity analysis , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[130]  Nam Seog Park.  Connectionist symbolic rule encoding using a generalized phase-locking mechanism , 2000, Expert Syst. J. Knowl. Eng..

[131]  D. Rumelhart,et al.  Philosophy and Connectionist Theory , 1991 .

[132]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[133]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[134]  Keith Stenning,et al.  Extension of the temporal synchrony approach to dynamic variable bindingin a connectionist inference system , 1995, Knowl. Based Syst..

[135]  Graeme S. Halford,et al.  Competing, or perhaps complementary, approaches to the dynamic-binding problem, with similar capacity limitations , 1993, Behavioral and Brain Sciences.

[136]  John E. Hummel,et al.  Distributed representations of structure: A theory of analogical access and mapping. , 1997 .

[137]  Geoffrey E. Hinton,et al.  A Distributed Connectionist Production System , 1988, Cogn. Sci..