Robotic roommates making pancakes

In this paper we report on a recent public experiment that shows two robots making pancakes using web instructions. In the experiment, the robots retrieve instructions for making pancakes from the World Wide Web and generate robot action plans from the instructions. This task is jointly performed by two autonomous robots: The first robot opens and closes cupboards and drawers, takes a pancake mix from the refrigerator, and hands it to the robot B. The second robot cooks and flips the pancakes, and then delivers them back to the first robot. While the robot plans in the scenario are all percept-guided, they are also limited in different ways and rely on manually implemented sub-plans for parts of the task. We will thus discuss the potential of the underlying technologies as well as the research challenges raised by the experiment.

[1]  Ian Horswill,et al.  Analysis of Adaptation and Environment , 1995, Artif. Intell..

[2]  Douglas B. Lenat,et al.  CYC: a large-scale investment in knowledge infrastructure , 1995, CACM.

[3]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[4]  Maria Fox,et al.  PDDL2.1: An Extension to PDDL for Expressing Temporal Planning Domains , 2003, J. Artif. Intell. Res..

[5]  Dan Klein,et al.  Accurate Unlexicalized Parsing , 2003, ACL.

[6]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[7]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[8]  E. Torres-Jara,et al.  Challenges for Robot Manipulation in Human Environments , 2006 .

[9]  Masayuki Inaba,et al.  Vision based behavior verification system of humanoid robot for daily environment tasks , 2006, 2006 6th IEEE-RAS International Conference on Humanoid Robots.

[10]  Charles C. Kemp,et al.  Challenges for robot manipulation in human environments [Grand Challenges of Robotics] , 2007, IEEE Robotics & Automation Magazine.

[11]  Nassir Navab,et al.  Edge-Based Template Matching with a Harmonic Deformation Model , 2008, VISIGRAPP.

[12]  Joris De Schutter,et al.  iTASC: a tool for multi-sensor integration in robot manipulation , 2008, 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[13]  Michael Beetz,et al.  Real-time perception-guided motion planning for a personal robot , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Moritz Tenorth,et al.  KNOWROB — knowledge processing for autonomous personal robots , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Dejan Pangercic,et al.  Real-time CAD model matching for mobile manipulation and grasping , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[16]  Leslie Pack Kaelbling,et al.  Hierarchical Planning in the Now , 2010, Bridging the Gap Between Task and Motion Planning.

[17]  Nassir Navab,et al.  Model globally, match locally: Efficient and robust 3D object recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Moritz Tenorth,et al.  Understanding and executing instructions for everyday manipulation tasks from the World Wide Web , 2010, 2010 IEEE International Conference on Robotics and Automation.

[19]  Moritz Tenorth,et al.  KNOWROB-MAP - knowledge-linked semantic object maps , 2010, 2010 10th IEEE-RAS International Conference on Humanoid Robots.

[20]  Moritz Tenorth,et al.  CRAM — A Cognitive Robot Abstract Machine for everyday manipulation in human environments , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  Michael Beetz,et al.  Becoming action-aware through reasoning about logged plan execution traces , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[22]  Michael Beetz,et al.  Simulation-based temporal projection of everyday robot object manipulation , 2011, AAMAS.

[23]  Leslie Pack Kaelbling,et al.  Hierarchical task and motion planning in the now , 2011, 2011 IEEE International Conference on Robotics and Automation.

[24]  Nico Blodow,et al.  Autonomous semantic mapping for robots performing everyday manipulation tasks in kitchen environments , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.