Neural associative memories for the integration of language, vision and action in an autonomous agent

Language understanding is a long-standing problem in computer science. However, the human brain is capable of processing complex languages with seemingly no difficulties. This paper shows a model for language understanding using biologically plausible neural networks composed of associative memories. The model is able to deal with ambiguities on the single word and grammatical level. The language system is embedded into a robot in order to demonstrate the correct semantical understanding of the input sentences by letting the robot perform corresponding actions. For that purpose, a simple neural action planning system has been combined with neural networks for visual object recognition and visual attention control mechanisms.

[1]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[2]  Günther Palm,et al.  Information Integration in a Multi-Stage Object Classifier , 2005, AMS.

[3]  Deb Roy,et al.  Grounded Situation Models for Robots: Where words and percepts meet , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  Sameer A. Nene,et al.  Columbia Object Image Library (COIL100) , 1996 .

[5]  Günther Palm,et al.  Word Recognition and Learning based on Associative Memories and Hidden Markov Models , 2008 .

[6]  Günther Palm,et al.  Word recognition and incremental learning based on neural associative memories and hidden Markov models , 2008, ESANN.

[7]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[8]  F. Pulvermüller.  The Neuroscience of Language , 2003 .

[9]  Günther Palm,et al.  Neural Associative Memories and Hidden Markov Models for Speech Recognition , 2007, 2007 International Joint Conference on Neural Networks.

[10]  G. Palm,et al.  Neural Networks for Visual Object Recognition Based on Selective Attention , 2005 .

[11]  S. Small.  The neuroscience of language , 2008, Brain and Language.

[12]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[13]  J. Hawkins,et al.  On Intelligence , 2004 .

[14]  G. Palm,et al.  On associative memory , 2004, Biological Cybernetics.

[15]  I C G Campbell,et al.  European Symposium on Artificial Neural Networks ESANN '95 , 1995 .

[16]  Günther Palm,et al.  Combining Visual Attention, Object Recognition and Associative Information Processing in a NeuroBotic System , 2005, Biomimetic Neural Learning for Intelligent Robots.

[17]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[18]  Prof. Dr. Valentino Braitenberg,et al.  Anatomy of the Cortex , 1991, Studies of Brain Function.

[19]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[20]  F. Attneave,et al.  The Organization of Behavior: A Neuropsychological Theory , 1949 .

[21]  W. M. Keck,et al.  Machine Psychology : Autonomous Behavior , Perceptual Categorization and Conditioning in a Brain-based Device , 2002 .

[22]  Yuehui Chen,et al.  Hierarchical Neural Networks , 2010 .

[23]  Günther Palm,et al.  Incremental learning in hierarchical neural networks for object recognition , 2005, ICINCO.

[24]  Gèunther Palm,et al.  Neural Assemblies: An Alternative Approach to Artificial Intelligence , 1982 .

[25]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[26]  Günther Palm,et al.  Modelling of syntactical processing in the cortex , 2007, Biosyst..

[27]  尚弘 島影.  National Institute of Standards and Technologyにおける超伝導研究及び生活 , 2001 .

[28]  Rebecca Fay,et al.  Feature selection and information fusion in hierarchical neural networks for iterative 3D-object recognition , 2007 .

[29]  Günther Palm,et al.  Biomimetic Neural Learning for Intelligent Robots - Intelligent Systems, Cognitive Robotics, and Neuroscience , 2005, Biomimetic Neural Learning for Intelligent Robots.

[30]  Jonathan G. Fiscus,et al.  Darpa Timit Acoustic-Phonetic Continuous Speech Corpus CD-ROM {TIMIT} | NIST , 1993 .

[31]  Robert Hecht-Nielsen,et al.  Confabulation theory , 2007, Scholarpedia.

[32]  Günther Palm,et al.  Hierarchical Neural Networks Utilising Dempster-Shafer Evidence Theory , 2006, ANNPR.

[33]  Deb Roy,et al.  RIPLEY , HAND ME THE CUP ! ( SENSORIMOTOR REPRESENTATIONS FOR GROUNDING WORD MEANING ) , 2003 .

[34]  T. M. Mayhew,et al.  Anatomy of the Cortex: Statistics and Geometry. , 1991 .