Following recipes with a cooking robot

In this thesis, we present BakeBot, a PR2 robot system that interprets natural language baking recipes into baking instructions which it follows to execute the recipe, from mise en place presentation of the ingredients through baking in a toaster oven. We developed parameterized motion primitives for baking. The motion primitives utilize the existing sensing and manipulation capabilities of the PR2 platform and also our new compliant control techniques to address environmental uncertainty. The system was first implemented as a static finite state machine, which was tested through 27 baking attempts, 16 of which successfully resulted in edible cookies. The system was then implemented as a dynamic state machine, in which the robot estimated the world state and planned sequences of motion primitives to follow the baking instructions inferred from the natural language recipe, which was tested thorough five baking attempts of two different recipes, all of which resulted in edible cookies. Thesis Supervisor: Daniela Rus Title: Professor of Electrical Engineering and Computer Science Mechanical Engineering Faculty Reader: John Leonard Title: Professor of Mechanical Engineering

[1]  Dieter Fox,et al.  Following directions using statistical machine translation , 2010, HRI 2010.

[2]  Jennifer Barry,et al.  Bakebot: Baking Cookies with the PR2 , 2011 .

[3]  Masayuki Inaba,et al.  Cooking for humanoid robot, a task that needs symbolic and geometric reasonings , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[4]  Neville Hogan,et al.  Impedance Control: An Approach to Manipulation , 1984, 1984 American Control Conference.

[5]  Maxim Likhachev,et al.  Search-based planning for manipulation with motion primitives , 2010, 2010 IEEE International Conference on Robotics and Automation.

[6]  Adam L. Berger,et al.  A Maximum Entropy Approach to Natural Language Processing , 1996, CL.

[7]  Stefan Schaal,et al.  Skill learning and task outcome prediction for manipulation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[8]  Matthew T. Mason,et al.  Compliance and Force Control for Computer Controlled Manipulators , 1981, IEEE Transactions on Systems, Man, and Cybernetics.

[9]  Matei T. Ciocarlie,et al.  Towards Reliable Grasping and Manipulation in Household Environments , 2010, ISER.

[10]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[11]  Matthew R. Walter,et al.  Approaching the Symbol Grounding Problem with Probabilistic Graphical Models , 2011, AI Mag..

[12]  Stefanie Tellex,et al.  Toward understanding natural language directions , 2010, HRI 2010.

[13]  Peter Corke,et al.  Autonomous aerial navigation and tracking of marine animals , 2011, ICRA 2011.

[14]  Dejan Pangercic,et al.  Robotic roommates making pancakes , 2011, 2011 11th IEEE-RAS International Conference on Humanoid Robots.

[15]  C. Stachniss,et al.  The PR 2 Workshop-Mobile Manipulation of Kitchen Containers , 2011 .

[16]  Bernhard Nebel,et al.  The FF Planning System: Fast Plan Generation Through Heuristic Search , 2011, J. Artif. Intell. Res..

[17]  Leslie Pack Kaelbling,et al.  Hierarchical Planning in the Now , 2010, Bridging the Gap Between Task and Motion Planning.

[18]  Charles C. Kemp,et al.  Challenges for robot manipulation in human environments [Grand Challenges of Robotics] , 2007, IEEE Robotics & Automation Magazine.

[19]  Michael Beetz,et al.  Real-time perception-guided motion planning for a personal robot , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Raymond J. Mooney,et al.  Learning to Interpret Natural Language Navigation Instructions from Observations , 2011, Proceedings of the AAAI Conference on Artificial Intelligence.

[21]  Stuart J. Russell,et al.  Combined Task and Motion Planning for Mobile Manipulation , 2010, ICAPS.

[22]  Dana S. Nau,et al.  SHOP2: An HTN Planning System , 2003, J. Artif. Intell. Res..