Action Categorisation in Multimodal Instructions

Situated agents interact both with their physical environment they are located in and with their conversational partners. As both the world and the language used in situated conversations are continuously changing, an agent must be able to adapt its grounded semantic representations by learning from new information. A pre-requisite for a dynamic, interactive approach to learning of grounded semantic representations is that an agent is equipped with a set of actions that define its strategies for identifying and connecting linguistic and perceptual information to its knowledge. In this talk we present our work on grounding spatial descriptions that argues that perceptual grounding is dynamic and adaptable to contexts. We describe a system called Kille which we use for interactive learning of objects and spatial relations from a human tutor. Finally, we describe our work on identifying interactive strategies of frame of reference assignment in spatial descriptions in a corpus of human-human dialogues and argue that there is no general preference for frame of reference assignment but this is linked to interaction strategies between agents that are adopted within a particular dialogue game.