Toward Interpreting Spatial Language Discourse with Grounding Graphs

In order to interact naturally with humans, robots must be able to engage in natural language dialog with their human teammate. To do so they need to model not only the state of the world but also the state of the interaction with the human user. In this paper we present an algorithm for understanding dialog in which information from multiple di alog turns is dynamically integrated into a probabilistic graphical model. Inference in the joint model enables the robot to find groundings in the world that correspond to the language across multiple dialog acts. To make inference efficient we model th e interaction state as a probabilistic model called Generali zed Grounding Graphs (G), which is dynamically instantiated from the natural language according to the hierarchical and compositional semantic structure of the utterances. Our previ ous work showed that a G model can find and execute plans corresponding to a single natural language command for mobi le manipulation. This paper extends the G model to support multi-turn dialog interaction and demonstrates its application on several examples; a full implementation and evaluation remain future work. Unifying the representations of a single command and a dialogue allows the training and inference algorithmsto be the same for both problems.