Building Plannable Representations with Mixed Reality

We propose Action-Oriented Semantic Maps (AOSMs), a representation that enables a robot to acquire object manipulation behaviors and semantic information about the environment from a human teacher with a Mixed Reality Head-Mounted Display (MR-HMD). AOSMs are a representation that captures both: a) high-level object manipulation actions in an object class’s local frame, and b) semantic representations of objects in the robot’s global map that are grounded for navigation. Humans can use a MR-HMD to teach the agent the information necessary for planning object manipulation and navigation actions by interacting with virtual 3D meshes overlaid on the physical workspace. We demonstrate that our system enables users to quickly and accurately teach a robot the knowledge required to autonomously plan and execute three household tasks: picking up a bottle and throwing it in the trash, closing a sink faucet, and flipping a light switch off.

[1]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[2]  Pavel Krsek,et al.  The Trimmed Iterative Closest Point algorithm , 2002, Object recognition supported by user interaction for service robots.

[3]  Yasushi Makihara,et al.  Object recognition supported by user interaction for service robots , 2002, Object recognition supported by user interaction for service robots.

[4]  Ben J. A. Kröse,et al.  BIRON, where are you? Enabling a robot to learn new places in a real home environment by integrating spoken dialog and visual localization , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  Dennis R. Wixon,et al.  Using the RITE method to improve products; a definition and a case study , 2007 .

[6]  Joachim Hertzberg,et al.  Towards semantic maps for mobile robots , 2008, Robotics Auton. Syst..

[7]  Andre Cohen,et al.  An object-oriented representation for efficient reinforcement learning , 2008, ICML '08.

[8]  Florentin Wörgötter,et al.  Cognitive agents - a procedural perspective relying on the predictability of Object-Action-Complexes (OACs) , 2009, Robotics Auton. Syst..

[9]  Mark Steedman,et al.  Object-Action Complexes: Grounded abstractions of sensory-motor processes , 2011, Robotics Auton. Syst..

[10]  Patric Jensfelt,et al.  Large-scale semantic mapping and reasoning with heterogeneous modalities , 2012, 2012 IEEE International Conference on Robotics and Automation.

[11]  Daniele Nardi,et al.  Knowledge Representation for Robots through Human-Robot Interaction , 2013, ICLP 2013.

[12]  Matthew R. Walter,et al.  Learning Semantic Maps from Natural Language Descriptions , 2013, Robotics: Science and Systems.

[13]  Jun Nakanishi,et al.  Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors , 2013, Neural Computation.

[14]  Antonios Gasteratos,et al.  Semantic mapping for mobile robotics tasks: A survey , 2015, Robotics Auton. Syst..

[15]  Marc Hanheide,et al.  Robot task planning and explanation in open and uncertain worlds , 2017, Artif. Intell..

[16]  David Whitney,et al.  Communicating Robot Arm Motion Intent Through Mixed Reality Head-mounted Displays , 2017, ISRR.

[17]  Gordon Cheng,et al.  Transferring skills to humanoid robots by extracting semantic representations from observations of human activities , 2017, Artif. Intell..

[18]  Frank Steinicke,et al.  Comparison of Multimodal Heading and Pointing Gestures for Co-Located Mixed Reality Human-Robot Interaction , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  Pieter Abbeel,et al.  Learning Plannable Representations with Causal InfoGAN , 2018, NeurIPS.

[20]  Yu Sun,et al.  A Survey of Knowledge Representation and Retrieval for Learning in Service Robotics , 2018, ArXiv.

[21]  Michael Beetz,et al.  Know Rob 2.0 — A 2nd Generation Knowledge Processing Framework for Cognition-Enabled Robotic Agents , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[22]  David Whitney,et al.  ROS Reality: A Virtual Reality Framework Using Consumer-Grade Hardware for ROS-Enabled Robots , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Subbarao Kambhampati,et al.  Projection-Aware Task Planning and Execution for Human-in-the-Loop Operation of Robots in a Mixed-Reality Workspace , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[24]  Jennifer Lee,et al.  Communicating Robot Motion Intent with Augmented Reality , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[25]  Ruediger Dillmann,et al.  Transparent Robot Behavior Using Augmented Reality in Close Human-Robot Interaction , 2019, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[26]  Stefanie Tellex,et al.  End-User Robot Programming Using Mixed Reality , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[27]  Stefanie Tellex,et al.  Flight, Camera, Action! Using Natural Language and Mixed Reality to Control a Drone , 2019, 2019 International Conference on Robotics and Automation (ICRA).