Using Spatial Semantic and Pragmatic Fields to Interpret Natural Language Pick-and-Place Instructions for a Mobile Service Robot

We present a methodology for enabling mobile service robots to follow natural language instructions for object pick-and-place tasks from non-expert users, with and without user-specified constraints, and with a particular focus on spatial language understanding. Our approach is capable of addressing both the semantic and pragmatic properties of object movement-oriented natural language instructions, and in particular, proposes a novel computational field representation for the incorporation of spatial pragmatic constraints in mobile manipulation task planning. The design and implementation details of our methodology are also presented, including the grammar utilized and our procedure for pruning multiple candidate parses based on context. The paper concludes with an evaluation of our approach implemented on a simulated mobile robot operating in both 2D and 3D home environments.

[1]  Maya Cakmak,et al.  Towards grounding concepts for transfer in goal learning from demonstration , 2011, 2011 IEEE International Conference on Development and Learning (ICDL).

[2]  Peter Ford Dominey,et al.  Anticipation and initiative in human-humanoid interaction , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[3]  Jenay M. Beer,et al.  The domesticated robot: Design guidelines for assisting older adults to age in place , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[4]  Hadas Kress-Gazit,et al.  Translating Structured English to Robot Controllers , 2008, Adv. Robotics.

[5]  Matthew R. Walter,et al.  Approaching the Symbol Grounding Problem with Probabilistic Graphical Models , 2011, AI Mag..

[6]  Marjorie Skubic,et al.  Spatial language for human-robot dialogs , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[7]  Luke S. Zettlemoyer,et al.  Learning to Parse Natural Language Commands to a Robot Control System , 2012, ISER.

[8]  Ioannis Iossifidis,et al.  Natural human-robot interaction through spatial language: A Dynamic Neural Field approach , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[9]  Monica N. Nicolescu,et al.  Natural methods for robot task learning: instructive demonstrations, generalization and practice , 2003, AAMAS '03.

[10]  John D. Kelleher,et al.  Towards a Cognitive System that Can Recognize Spatial Regions Based on Context , 2012, AAAI.

[11]  Deb K. Roy,et al.  Learning visually grounded words and syntax for a scene description task , 2002, Comput. Speech Lang..

[12]  B. Landau,et al.  “What” and “where” in spatial language and spatial cognition , 1993 .

[13]  J O'Keefe,et al.  Vector grammar, places, and the functional role of the spatial prepositions in English , 2001 .

[14]  Maja J. Mataric,et al.  Modeling dynamic spatial relations with global properties for natural language-based human-robot interaction , 2013, 2013 IEEE RO-MAN.

[15]  Rachid Alami,et al.  Situation assessment for human-robot interactive object manipulation , 2011, 2011 RO-MAN.

[16]  Maja J. Mataric,et al.  Using semantic fields to model dynamic spatial relations in a robot architecture for natural language instruction of service robots , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Manuela M. Veloso,et al.  Using dialog and human observations to dictate tasks to a learning robot assistant , 2008, Intell. Serv. Robotics.

[18]  Matthias Scheutz,et al.  Learning actions from human-robot dialogues , 2011, 2011 RO-MAN.

[19]  Stefanie Tellex,et al.  Toward understanding natural language directions , 2010, HRI 2010.

[20]  Laura A. Carlson,et al.  Formulating Spatial Descriptions across Various Dialogue Contexts , 2009, Spatial Language and Dialogue.

[21]  John D. Kelleher,et al.  Mediating between Qualitative and Quantitative Representations for Task-Orientated Human-Robot Interaction , 2007, IJCAI.

[22]  John D. Kelleher,et al.  Applying Computational Models of Spatial Prepositions to Visually Situated Dialog , 2009, CL.