A neural-network architecture for syntax analysis

Artificial neural networks (ANN's), due to their inherent parallelism, offer an attractive paradigm for implementation of symbol processing systems for applications in computer science and artificial intelligence. This paper explores systematic synthesis of modular neural-network architectures for syntax analysis using a prespecified grammar--a prototypical symbol processing task which finds applications in programming language interpretation, syntax analysis of symbolic expressions, and high-performance compilers. The proposed architecture is assembled from ANN components for lexical analysis, stack, parsing and parse tree construction. Each of these modules takes advantage of parallel content-based pattern matching using a neural associative memory. The proposed neural-network architecture for syntax analysis provides a relatively efficient and high performance alternative to current computer systems for applications that involve parsing of LR grammars which constitute a widely used subset of deterministic context-free grammars. Comparison of quantitatively estimated performance of such a system [implemented using current CMOS very large scale integration (VLSI) technology] with that of conventional computers demonstrates the benefits of massively parallel neural-network architectures for symbol processing applications.

[1]  Alex M. Andrew,et al.  Intelligent Hybrid Systems , 1999 .

[2]  Branko Soucek.  Neural and intelligent systems integration - fifth and sixth generation integrated reasoning information systems , 1991, Sixth-generation computer technology series.

[3]  Padhraic Smyth,et al.  Discrete recurrent neural networks for grammatical inference , 1994, IEEE Trans. Neural Networks.

[4]  Bruce J. MacLennan,et al.  Principles of Programming Languages , 1983 .

[5]  Charles P. Dolan,et al.  Tensor Product Production System: a Modular Architecture and Representation , 1989 .

[6]  Chris Mellish,et al.  Natural Language Processing in Pop-11: An Introduction to Computational Linguistics , 1989 .

[7]  Vasant Honavar,et al.  Toward Learning Systems That Integrate Different Strategies and Representations , 1993 .

[8]  L. Shastri,et al.  A Connectionist System for Rule Based Reasoning With Multi-Place Predicates and Variables , 1989 .

[9]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[10]  Nigel P. Chapman,et al.  LR Parsing: Theory and Practice , 1988 .

[11]  Lloyd W. Massengill,et al.  An analog neural hardware implementation using charge-injection multipliers and neutron-specific gain control , 1992, IEEE Trans. Neural Networks.

[12]  Jude Shavlik,et al.  A Framework for Combining Symbolic and Neural Learning , 1992 .

[13]  Giovanni Soda,et al.  Unified Integration of Explicit Knowledge and Learning by Example in Recurrent Networks , 1995, IEEE Trans. Knowl. Data Eng..

[14]  H.P. Graf,et al.  A reconfigurable CMOS neural network , 1990, 1990 37th IEEE International Conference on Solid-State Circuits.

[15]  Peter Kogge,et al.  VLSI and rule-based systems , 1988, CARN.

[16]  George Berg,et al.  A Connectionist Parser with Recursive Sentence Structure and Lexical Disambiguation , 1992, AAAI.

[17]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[18]  Ilié Popescu.  Hierarchical neural network for rules control in knowledge-based expert systems , 1995, Neural Parallel Sci. Comput..

[19]  James L. McClelland,et al.  Reflections on Cognition and Parallel Distributed Processing , 1987 .

[20]  Krste Asanovic,et al.  SPACE: symbolic processing in associative computing elements , 1995 .

[21]  H. Samueli,et al.  A 200 MHz CMOS pipelined multiplier-accumulator using a quasi-domino dynamic full-adder cell design , 1993 .

[22]  C. Lee Giles,et al.  Using Prior Knowledge in a {NNPDA} to Learn Context-Free Languages , 1992, NIPS.

[23]  Mark A. Fanty,et al.  Context-free parsing with connectionist networks , 1987 .

[24]  Alex Waibel,et al.  PARSEC: a structured connectionist parsing system for spoken language , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[25]  Raymond Glover,et al.  A distributed parallel associative processor (DPAP) for the execution of logic programs , 1995 .

[26]  Chilukuri K. Mohan,et al.  Fault Tolerance of Neural Networks , 1994 .

[27]  Rajesh Parekh,et al.  A Polynomial Time Incremental Algorithm for Regular Grammar Inference , 1997 .

[28]  Osamu Saito,et al.  An 8 G connections-per-second 54 mW digital neural network chip low-power chain-reaction architecture , 1992, 1992 IEEE International Solid-State Circuits Conference Digest of Technical Papers.

[29]  R. C. Lacher,et al.  Hierarchical Architectures for Reasoning , 1995 .

[30]  Alfred V. Aho,et al.  Principles of Compiler Design , 1977 .

[31]  Gadi Pinkas.  A Fault Tolerant Connectionist Architecture for Construction of Logic Proofs , 1993 .

[32]  Jerzy B. Lont.  Analog CMOS implementation of a multi-layer perceptron with nonlinear synapses , 1992, IEEE Trans. Neural Networks.

[33]  S C Kleene,et al.  Representation of Events in Nerve Nets and Finite Automata , 1951 .

[34]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[35]  Ian Robinson.  The Pattern Addressable Memory: Hardware for Associative Processing , 1989 .

[36]  Professor Seppo Sippu,et al.  Parsing Theory , 1988, EATCS Monographs on Theoretical Computer Science.

[37]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[38]  W. Schneider.  Connectionism: Is it a paradigm shift for psychology? , 1987 .

[39]  Vasant Honavar,et al.  Symbolic Artificial Intelligence and Numeric Artificial Neural Networks: Towards A Resolution of the Dichotomy , 1995 .

[40]  G.C.N. Yeung,et al.  Principles of programming languages: Design, evaluation, and implementation , 1985, Proceedings of the IEEE.

[41]  Robert B. Allen,et al.  Connectionist Language Users , 1990 .

[42]  Yan Ng,et al.  Unify with Active Memory , 1989 .

[43]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[44]  C. Lee Giles,et al.  Learning a class of large finite state machines with a recurrent neural network , 1995, Neural Networks.

[45]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[46]  Vasant Honavar,et al.  Books-Received - Artificial Intelligence and Neural Networks - Steps Toward Principled Integration , 1994 .

[47]  Donald A. Norman,et al.  Reflections on cognition and parallel distributed processing , 1986 .

[48]  Takashi Kimura,et al.  High-Speed CAM-Based Architecture for a Prolog Machine (ASCA) , 1988, IEEE Trans. Computers.

[49]  T. R. Damarla,et al.  Fault tolerance of neural networks , 1989, Proceedings. IEEE Energy and Information Technologies in the Southeast'.

[50]  C. H. Sequin,et al.  Fault tolerance in artificial neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[51]  M.H. Hassoun,et al.  Fundamentals of Artificial Neural Networks , 1996, Proceedings of the IEEE.

[52]  C. Lee Giles,et al.  Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants , 1996, Neural Computation.

[53]  Raymond L. Watrous,et al.  Induction of Finite-State Languages Using Second-Order Recurrent Networks , 1992, Neural Computation.

[54]  M. Henning,et al.  An Architecture for Associative Processing of Large Knowledge Bases (LKBs) , 1987, Comput. J..

[55]  Edgar Sánchez-Sinencio,et al.  A modular CMOS design of a Hamming network , 1992, IEEE Trans. Neural Networks.

[56]  Eduardo Sontag,et al.  Turing computability with neural nets , 1991 .

[57]  Kenneth J. Thurber,et al.  Associative and Parallel Processors , 1975, CSUR.

[58]  Risto Miikkulainen,et al.  Subsymbolic Parsing of Embedded Structures , 1995 .

[59]  Laurent Miclet,et al.  Structural Methods in Pattern Recognition , 1986 .

[60]  Michael C. Mozer,et al.  Discovering the Structure of a Reactive Environment by Exploration , 1990, Neural Computation.

[61]  Michael G. Dyer,et al.  Connectionist Natural Language Processing: A Status Report , 1995 .

[62]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[63]  Alberto Sanfeliu,et al.  Understanding Neural Networks for Grammatical Inference and Recognition , 1993 .

[64]  R. Peter,et al.  Recursive Functions in Computer Theory , 1982 .

[65]  R. Kumar.  NCMOS: a high performance CMOS logic , 1994 .

[66]  Rajesh Parekh,et al.  Automata Induction, Grammar Inference, and Language Acquisition , 2000 .

[67]  Derick Wood,et al.  Theory of computation , 1986 .

[68]  Koichi Wada,et al.  A Hardware Syntactic Analysis Processor , 1987, IEEE Micro.

[69]  Alan F. Murray,et al.  Integrated pulse stream neural networks: results, issues, and pointers , 1992, IEEE Trans. Neural Networks.

[70]  Mona E. Zaghloul,et al.  VLSI implementation of synaptic weighting and summing in pulse coded neural-type cells , 1992, IEEE Trans. Neural Networks.

[71]  Lawrence A. Bookman.  A Framework for Integrating Relational and Associational Knowledge for Comprehension , 1995 .

[72]  Vasant Honavar,et al.  A Neural Network Architecture for High-Speed Database Query Processing , 1995 .

[73]  Bing J. Sheu,et al.  Design and characterization of analog VLSI neural network modules , 1993 .

[74]  Nikola Kasabov,et al.  Hardware support for data parallelism in production systems , 1995 .

[75]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[76]  Michael C. Mozer,et al.  A Connectionist Symbol Manipulator that Discovers the Structure of Context-Free Languages , 1992, NIPS.

[77]  Pat Langley,et al.  Elements of Machine Learning , 1995 .

[78]  Anoop Gupta.  Parallelism in production systems , 1987 .

[79]  Vasant Honavar,et al.  Integrating Symbol Processing Systems and Connectionist Networks , 1995 .

[80]  Alfred V. Aho,et al.  Principles of Compiler Design (Addison-Wesley series in computer science and information processing) , 1977 .

[81]  W. Omlin.  Fault-tolerant Implementation of Finite-state Automata in Recurrent Neural Networks , 1995 .

[82]  Frank van der Velde.  Symbol manipulation with neural networks : production of a context-free language using a modifiable working memory , 1995 .

[83]  Vasant Honavar,et al.  Coordination and control structures and processes: possibilities for connectionist networks (CN) , 1990, J. Exp. Theor. Artif. Intell..

[84]  H. Wallinga,et al.  70 input, 20 nanosecond pattern classifier , 1993, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[85]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[86]  Takeshi Sakata,et al.  A single 1.5-V digital chip for a 106 synapse neural network , 1993, IEEE Trans. Neural Networks.

[87]  R. B. Allen,et al.  Identifying and discriminating temporal events with connectionist language users , 1989 .

[88]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[89]  I. Noda,et al.  A learning method for recurrent networks based on minimization of finite automata , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[90]  D. C. Cooper,et al.  Theory of Recursive Functions and Effective Computability , 1969, The Mathematical Gazette.

[91]  John Taylor,et al.  Design, implementation and evaluation of a high-speed integrated Hamming neural classifier , 1994 .

[92]  D. Schulenburg,et al.  Sentence processing with realistic feedback , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[93]  Vasant Honavar,et al.  A Neural Architecture for Content as well as Address-based Storage and Recall: Theory and Applications , 1995, Connect. Sci..

[94]  James A. Hendler.  Beyond the Fifth Generation: parallel AI research in Japan , 1994, IEEE Expert.

[95]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.