Understanding User Instructions by Utilizing Open Knowledge for Service Robots

Understanding user instructions in natural language is an active research topic in AI and robotics. Typically, natural user instructions are high-level and can be reduced into low-level tasks expressed in common verbs (e.g., `take', `get', `put'). For robots understanding such instructions, one of the key challenges is to process high-level user instructions and achieve the specified tasks with robots' primitive actions. To address this, we propose novel algorithms by utilizing semantic roles of common verbs defined in semantic dictionaries and integrating multiple open knowledge to generate task plans. Specifically, we present a new method for matching and recovering semantics of user instructions and a novel task planner that exploits functional knowledge of robot's action model. To verify and evaluate our approach, we implemented a prototype system using knowledge from several open resources. Experiments on our system confirmed the correctness and efficiency of our algorithms. Notably, our system has been deployed in the KeJia robot, which participated the annual RoboCup@Home competitions in the past three years and achieved encouragingly high scores in the benchmark tests.

[1]  Christopher D. Manning,et al.  Generating Typed Dependency Parses from Phrase Structure Parses , 2006, LREC.

[2]  Moritz Tenorth,et al.  KnowRob: A knowledge processing infrastructure for cognition-enabled robots , 2013, Int. J. Robotics Res..

[3]  Stefanie Tellex,et al.  Toward understanding natural language directions , 2010, 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[4]  Michael Beetz,et al.  Everything robots always wanted to know about housework (but were afraid to ask) , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  C. Shah,et al.  Building Plans for Household Tasks from Distributed Knowledge , 2022 .

[6]  Xiaoping Chen,et al.  Handling Open Knowledge for Service Robots , 2013, IJCAI.

[7]  Hema Swetha Koppula,et al.  RoboBrain: Large-Scale Knowledge Engine for Robots , 2014, ArXiv.

[8]  John B. Lowe,et al.  The Berkeley FrameNet Project , 1998, ACL.

[9]  Rakesh Gupta,et al.  Common Sense Data Acquisition for Indoor Mobile Robots , 2004, AAAI.

[10]  Christopher D. Manning,et al.  The Stanford Typed Dependencies Representation , 2008, CF+CDPE@COLING.

[11]  Matthias Scheutz,et al.  What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution , 2009, 2009 IEEE International Conference on Robotics and Automation.

[12]  Hadas Kress-Gazit,et al.  Temporal-Logic-Based Reactive Mission and Motion Planning , 2009, IEEE Transactions on Robotics.

[13]  Xiaoping Chen,et al.  Developing high-level cognitive functions for service robots , 2010, AAMAS.

[14]  Feng Wu,et al.  Towards a Principled Solution to Simulated Robot Soccer , 2012, RoboCup.

[15]  Manuela M. Veloso,et al.  Learning environmental knowledge from task-based human-robot dialog , 2013, 2013 IEEE International Conference on Robotics and Automation.

[16]  Moritz Tenorth,et al.  KNOWROB-MAP - knowledge-linked semantic object maps , 2010, 2010 10th IEEE-RAS International Conference on Humanoid Robots.

[17]  Matthew R. Walter,et al.  Learning spatial-semantic representations from natural language descriptions and scene classifications , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Ashutosh Saxena,et al.  Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions , 2014, Int. J. Robotics Res..

[19]  Markus Vincze,et al.  AfRob: The affordance network ontology for robots , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Xiaoping Chen,et al.  Understanding Instructions on Large Scale for Human-Robot Interaction , 2014, 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT).

[21]  Xiaoping Chen,et al.  Multi-mode Natural Language Processing for human-robot interaction , 2015, Web Intell..

[22]  Moritz Tenorth,et al.  Putting People's Common Sense into Knowledge Bases of Household Robots , 2010, KI.

[23]  Séverin Lemaignan,et al.  Grounding the Interaction: Knowledge Management for Interactive Robots , 2013, KI - Künstliche Intelligenz.

[24]  Martin Gebser,et al.  Engineering an Incremental ASP Solver , 2008, ICLP.

[25]  Matthias Scheutz,et al.  Grounding Natural Language References to Unvisited and Hypothetical Locations , 2013, AAAI.

[26]  Matthias Scheutz,et al.  Robust spoken instruction understanding for HRI , 2010, 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[27]  Matthew R. Walter,et al.  Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation , 2011, AAAI.

[28]  Paul Bogaards,et al.  Dictionaries for Learners of English , 1996 .

[29]  Xiaoping Chen,et al.  Toward open knowledge enabling for human-robot interaction , 2013, HRI 2013.

[30]  Michael Beetz,et al.  Grounding the Interaction: Anchoring Situated Discourse in Everyday Human-Robot Interaction , 2012, Int. J. Soc. Robotics.

[31]  Michael A. West,et al.  A general service list of English words, with semantic frequencies and a supplementary word-list for the writing of popular science and technology , 1953 .

[32]  Kevin Lee,et al.  Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions , 2014, Int. J. Robotics Res..