Assembling Furniture by Asking for Help from a Human Partner

Robots inevitably fail, often without the ability to recover autonomously. A human partner can help a robot recover from a failure, but it is challenging even for a willing and motivated helper to determine what actions will effectively aid the robot. To address this problem, we demonstrate an approach for enabling a robot to communicate its need for specific help to a human partner using language. Our approach automatically detects failures, then generates targeted spoken-language request such as “Hand me the white table leg.” Once the human partner has repaired the failure condition, the system resumes full autonomy. We present a novel algorithm for generating help requests by emulating the human’s ability to interpret a command, based on the Generalized Grounding Graph framework. We report preliminary results for an early version of our algorithm.

[1]  井上 博允,et al.  Force Feedback in Precise Assembly Tasks , 1975 .

[2]  Michael L. Geis,et al.  Syntax and Semantics. Volume 3 : Speech Acts , 1976 .

[3]  Thomas L. DeFazio,et al.  Simplified generation of all mechanical assembly sequences , 1987, IEEE Journal on Robotics and Automation.

[4]  Arthur C. Sanderson,et al.  A correct and complete algorithm for the generation of mechanical assembly sequences , 1989, Proceedings, 1989 International Conference on Robotics and Automation.

[5]  Randall H. Wilson.  Minimizing user queries in interactive assembly planning , 1993, [1993] Proceedings IEEE International Conference on Robotics and Automation.

[6]  Tomoichi Takahashi,et al.  Robotic assembly operation teaching in a virtual environment , 1994, IEEE Trans. Robotics Autom..

[7]  Jean-Claude Latombe,et al.  Geometric Reasoning About Mechanical Assembly , 1994, Artif. Intell..

[8]  David Kortenkamp,et al.  Adjustable Autonomy for Human-Centered Autonomous Systems on Mars , 1998 .

[9]  James H. Martin,et al.  Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 2nd Edition , 2000, Prentice Hall series in artificial intelligence.

[10]  Bruno Siciliano,et al.  Achieving a cooperative behavior in a dual-arm robot system via a modular control structure , 2001, J. Field Robotics.

[11]  Bernhard Nebel,et al.  The FF Planning System: Fast Plan Generation Through Heuristic Search , 2011, J. Artif. Intell. Res..

[12]  Ehud Reiter,et al.  Book Reviews: Building Natural Language Generation Systems , 2000, CL.

[13]  William Whittaker,et al.  Skyworker: a robot for assembly, inspection and maintenance of large scale orbital facilities , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[14]  W. Doggett,et al.  Robotic assembly of truss structures for space systems and future research plans , 2002, Proceedings, IEEE Aerospace Conference.

[15]  D. Roy.  A TRAINABLE VISUALLY-GROUNDED SPOKEN LANGUAGE GENERATION SYSTEM , 2002 .

[16]  Terrence Fong,et al.  Robot, asker of questions , 2003, Robotics Auton. Syst..

[17]  Kazuo Machida,et al.  Robot control strategy for in-orbit assembly of a micro satellite , 2004, Adv. Robotics.

[18]  Brett Kennedy,et al.  HETEROGENEOUS ROBOTIC SYSTEMS FOR ASSEMBLY AND SERVICING , 2005 .

[19]  Benjamin Kuipers,et al.  Walk the Talk: Connecting Language, Knowledge, and Action in Route Instructions , 2006, AAAI.

[20]  Frederik W. Heger,et al.  Human-Robot Teams for Large-Scale Assembly , 2007 .

[21]  Matthias Scheutz,et al.  What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution , 2009, 2009 IEEE International Conference on Robotics and Automation.

[22]  Raymond J. Mooney,et al.  Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language , 2014, J. Artif. Intell. Res..

[23]  Dan Klein,et al.  A Game-Theoretic Approach to Generating Spatial Descriptions , 2010, EMNLP.

[24]  Stefanie Tellex,et al.  Toward understanding natural language directions , 2010, 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[25]  Pieter Abbeel,et al.  Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding , 2010, 2010 IEEE International Conference on Robotics and Automation.

[26]  Mariët Theune,et al.  Report on the Second Second Challenge on Generating Instructions in Virtual Environments (GIVE-2.5) , 2011, ENLG.

[27]  Matthew R. Walter,et al.  Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation , 2011, AAAI.

[28]  Alexander Koller,et al.  Combining symbolic and corpus-based approaches for the generation of successful referring expressions , 2011, ENLG.

[29]  Juan Rojas,et al.  Analysis of autonomous cooperative assembly using coordination schemes by heterogeneous robots using a control basis approach , 2012, Autonomous Robots.

[30]  Luke S. Zettlemoyer,et al.  A Joint Model of Language and Perception for Grounded Attribute Learning , 2012, ICML.

[31]  Stefanie Tellex,et al.  Interpreting and Executing Recipes with a Cooking Robot , 2012, ISER.