Multi-network system for sensory integration

The aim of our work is to provide a better understanding of multisensory interactions. Starting from an hypothesis of cognitive psychology, we propose a model of multimodal associative memory that integrates all the modality-specific information. The modular architecture consists of different neutral networks that cooperate for modeling both modality-specific low-level recognition and multi-modal high-level identification. A version with three perceptive modalities has been implemented and tested. Experiments confirm the validity of hypothesis on the functional architecture, since the model can simulate a good identification even if one or two modalities are not available. Other realistic phenomena can be observed on the model, such as evocation of mental images or a behavior similar to the McGurk effect.

[1]  Didier Puzenat,et al.  A multisensory identification system for robotics , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[2]  Risto Miikkulainen,et al.  Convergence-Zone Episodic Memory: Analysis and Simulations , 1997, Neural Networks.

[3]  Heekuck Oh,et al.  Adaptation of the relaxation method for learning in bidirectional associative memory , 1994, IEEE Trans. Neural Networks.

[4]  Naohiro Ishii,et al.  A self-supervised learning system for pattern recognition by sensory integration , 1999, Neural Networks.

[5]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[6]  J. Eich.  A composite holographic associative recall model. , 1982 .

[7]  B Kosko,et al.  Adaptive bidirectional associative memories. , 1987, Applied optics.

[8]  Stéphane Rousset,et al.  Facenet: A connectionist model of face identification in context. , 1991 .

[9]  E. Tulving.  Elements of episodic memory , 1983 .

[10]  Terrence J. Sejnowski,et al.  Neural network models of sensory integration for improved vowel recognition , 1990, Proc. IEEE.

[11]  Douglas L. Hintzman,et al.  "Schema Abstraction" in a Multiple-Trace Memory Model , 1986 .

[12]  B. Humpert,et al.  Bidirectional Associative Memory for Three Patterns , 1990 .

[13]  John E. Markel,et al.  Linear Prediction of Speech , 1976, Communication and Cybernetics.

[14]  S Edelman,et al.  Representation is representation of similarities , 1996, Behavioral and Brain Sciences.

[15]  Hélène Paugam-Moisy,et al.  A computational model for binding sensory modalities , 2000 .

[16]  Didier Puzenat.  Priming an Artificial Neural Classifier , 1995, IWANN.