Coupling perception and simulation: steps towards conversational robotics

Human cognition makes extensive use of visualization and imagination. As a first step towards giving a robot similar abilities, we have built a robotic system that uses a perceptually-coupled physical simulator to produce an internal world model of the robot's environment. Real-time perceptual coupling ensures that the model is constantly kept in synchronization with the physical environment as the robot moves and obtains new sense data. This model allows the robot to be aware of objects no longer in its field of view (a form of "object permanence"), as well as to visualize its environment through the eyes of the user by enabling virtual shifts in point of view using synthetic vision operating within the simulator. This architecture provides a basis for our long term goals of developing conversational robots that can ground the meaning of spoken language in terms of sensorimotor representations.

[1]  Karen Wynn,et al.  Limits to Infants' Knowledge of Objects: The Case of Magical Appearance , 1998 .

[2]  S. Kosslyn,et al.  Neural foundations of imagery , 2001, Nature Reviews Neuroscience.

[3]  J. Lammens.  A computational model of color perception and color naming , 1995 .

[4]  E Bizzi,et al.  Motor learning through the combination of primitives. , 2000, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[5]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[6]  Jeffrey Mark Siskind,et al.  Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic , 1999, J. Artif. Intell. Res..

[7]  J. Feldman,et al.  Karma: knowledge-based active representations for metaphor and aspect , 1997 .

[8]  Laura A. Carlson,et al.  Grounding spatial language in perception: an empirical and computational investigation. , 2001, Journal of experimental psychology. General.

[9]  Helge J. Ritter,et al.  Multi-modal human-machine communication for instructing robot grasping tasks , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Jay G. Wilpon,et al.  SAM: a perceptive spoken language-understanding robot , 1992, IEEE Trans. Syst. Man Cybern..

[11]  Deb Roy,et al.  Grounded Semantic Composition for Visual Scenes , 2011, J. Artif. Intell. Res..

[12]  P. Bloom.  Mindreading, Communication and the Learning of Names for Things , 2002 .

[13]  Deb Roy,et al.  A trainable spoken language understanding system for visual object selection , 2002, INTERSPEECH.

[14]  B. Achiriloaie,et al.  VI REFERENCES , 1961 .

[15]  Jerry Pratt,et al.  Series elastic actuators for high fidelity force control , 2002 .

[16]  Jerome A. Feldman,et al.  When push comes to shove: a computational model of the role of motor control in the acquisition of action verbs , 1997 .

[17]  Brian Cantwell Smith,et al.  On the origin of objects , 1997, Trends in Cognitive Sciences.

[18]  Luís Seabra Lopes,et al.  Semisentient robots: routes to integrated intelligence , 2001 .

[19]  Deb K. Roy,et al.  Learning visually grounded words and syntax for a scene description task , 2002, Comput. Speech Lang..

[20]  Patrick Suppes,et al.  Language and Learning for Robots , 1994 .

[21]  Deb Roy,et al.  Grounded spoken language acquisition: experiments in word learning , 2003, IEEE Trans. Multim..