Herb 2.0: Lessons Learned From Developing a Mobile Manipulator for the Home

We present the hardware design, software architecture, and core algorithms of Herb 2.0, a bimanual mobile manipulator developed at the Personal Robotics Lab at Carnegie Mellon University, Pittsburgh, PA. We have developed Herb 2.0 to perform useful tasks for and with people in human environments. We exploit two key paradigms in human environments: that they have structure that a robot can learn, adapt and exploit, and that they demand general-purpose capability in robotic systems. In this paper, we reveal some of the structure present in everyday environments that we have been able to harness for manipulation and interaction, comment on the particular challenges on working in human spaces, and describe some of the lessons we learned from extensively testing our integrated platform in kitchen and office environments.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  Nils J. Nilsson,et al.  A mobius automation: an application of artificial intelligence techniques , 1969, IJCAI 1969.

[3]  Harry G. Barrow,et al.  A Versatile Computer-Controlled Assembly System , 1973, IJCAI.

[4]  Harry G. Barrow,et al.  A Versatile System for Computer-Controlled Assembly , 1975, Artif. Intell..

[5]  Alan M. Thompson.  The Navigation System of the JPL Robot , 1977, IJCAI.

[6]  Russell H. Taylor,et al.  Automatic Synthesis of Fine-Motion Strategies for Robots , 1984 .

[7]  W. Eric L. Grimson,et al.  Handey: A robot system that recognizes, plans, and manipulates , 1987, Proceedings. 1987 IEEE International Conference on Robotics and Automation.

[8]  A. Ruina,et al.  Planar sliding with dry friction Part 1. Limit surface and moment function , 1991 .

[9]  Hikaru Inooka,et al.  Hand-over of an object between human and robot , 1992, [1992] Proceedings IEEE International Workshop on Robot and Human Communication.

[10]  Patrick A. O'Donnell,et al.  HANDEY: A Robot Task Planner , 1992 .

[11]  Jean-Claude Latombe,et al.  Planning motions with intentions , 1994, SIGGRAPH.

[12]  Yizong Cheng,et al.  Mean Shift, Mode Seeking, and Clustering , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Hikaru Inooka,et al.  Motion planning for hand-over between human and robot , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[14]  Rachid Alami,et al.  A General Framework For Multi-Robot Cooperation and Its Implementation on a Set of Three Hilare Robots , 1995, ISER.

[15]  Oussama Khatib,et al.  Coordination and decentralized cooperation of multiple mobile manipulators , 1996, J. Field Robotics.

[16]  Kazuo Tanie,et al.  Human interaction with a service robot: mobile-manipulator handing over an object to a human , 1997, Proceedings of International Conference on Robotics and Automation.

[17]  Kurt Konolige.  COLBERT: A Language for Reactive Control in Sapphira , 1997, KI.

[18]  Hideo Tsukune,et al.  Delivery by hand between human and robot based on fingertip force-torque information , 1998, Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No.98CH36190).

[19]  Masayuki Inaba,et al.  Design and development of research platform for perception-action integration in humanoid robot: H6 , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[20]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[21]  Lydia E. Kavraki,et al.  Randomized path planning for linkages with closed kinematic chains , 2001, IEEE Trans. Robotics Autom..

[22]  Kikuo Fujimura,et al.  The intelligent ASIMO: system overview and integration , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[23]  Masayuki Inaba,et al.  Online 3D vision, motion planning and bipedal locomotion control coupling system of humanoid robot: H7 , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[24]  A. ADoefaa,et al.  ? ? ? ? f ? ? ? ? ? , 2003 .

[25]  Roberto Ierusalimschy,et al.  Programming in Lua , 2003 .

[26]  Thierry Siméon,et al.  Sampling-Based Motion Planning under Kinematic Loop-Closure Constraints , 2004, WAFR.

[27]  Clark F. Olson,et al.  Efficient Pose Clustering Using a Randomized Algorithm , 1997, International Journal of Computer Vision.

[28]  Katsu Yamane,et al.  Synthesizing animations of human manipulation tasks , 2004, SIGGRAPH 2004.

[29]  Kamal K. Gupta,et al.  Path planning with general end-effector constraints: using task space to guide configuration space search , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[30]  Shuuji Kajita,et al.  Development of humanoid robot HRP-3P , 2005, 5th IEEE-RAS International Conference on Humanoid Robots, 2005..

[31]  J. Andrew Bagnell,et al.  Maximum margin planning , 2006, ICML.

[32]  Alonzo Kelly,et al.  Toward Reliable Off Road Autonomous Vehicles Operating in Challenging Environments , 2006, Int. J. Robotics Res..

[33]  Matthias Jüngel,et al.  XABSL - A Pragmatic Approach to Behavior Engineering , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[34]  Victor Ng-Thow-Hing,et al.  Toward Interactive Reaching in Static Environments for Humanoid Robots , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[35]  Rachid Alami,et al.  Grasp Planning for Interactive Object Manipulation , 2006 .

[36]  Tamim Asfour,et al.  An integrated approach to inverse kinematics and path planning for redundant manipulators , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[37]  Alin Albu-Schäffer,et al.  The DLR lightweight robot: design and control concepts for robots in human environments , 2007, Ind. Robot.

[38]  Rachid Alami,et al.  Spatial reasoning for human robot interaction , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Charles C. Kemp,et al.  Human-Robot Interaction for Cooperative Manipulation: Handing Objects to One Another , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[40]  Rachid Alami,et al.  Exploratory Study of a Robot Approaching a Person in the Context of Handing Over an Object , 2007, AAAI Spring Symposium: Multidisciplinary Collaboration for Socially Assistive Robotics.

[41]  Thomas Allen,et al.  A Planning System for Autonomous Ground Vehicles Operating in Unstructured Dynamic Environments , 2007 .

[42]  Alonzo Kelly,et al.  Toward Optimal Sampling in the Space of Paths , 2007, ISRR.

[43]  Andrew Y. Ng,et al.  STAIR: Hardware and Software Architecture , 2007 .

[44]  Mike Stilman,et al.  Task constrained motion planning in robot joint space , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[45]  Siddhartha S. Srinivasa,et al.  The robotic busboy: Steps towards developing a mobile robotic home assistant , 2008 .

[46]  R. Dillmann,et al.  Design of the planner of team AnnieWAY’s autonomous vehicle used in the DARPA Urban Challenge 2007 , 2008, 2008 IEEE Intelligent Vehicles Symposium.

[47]  Julius Ziegler,et al.  Team AnnieWAY's autonomous system for the 2007 DARPA Urban Challenge , 2008, J. Field Robotics.

[48]  Kenji KANEKO,et al.  Humanoid robot HRP-3 , 2004, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Alois Knoll,et al.  Human-robot interaction in handing-over tasks , 2008, RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication.

[50]  Siddhartha S. Srinivasa,et al.  CHOMP: Gradient optimization techniques for efficient motion planning , 2009, 2009 IEEE International Conference on Robotics and Automation.

[51]  Siddhartha S. Srinivasa,et al.  Generality and Simple Hands , 2009, ISRR.

[52]  Tiffany L. Chen,et al.  Hand it over or set it down: A user study of object delivery with an assistive mobile manipulator , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[53]  Siddhartha S. Srinivasa,et al.  Pose-constrained whole-body planning using Task Space Region Chains , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[54]  Alexander Ferrein,et al.  A Lua-based Behavior Engine for Controlling the Humanoid Robot Nao , 2009, RoboCup.

[55]  Ross A. Knepper,et al.  Path diversity is only part of the problem , 2009, 2009 IEEE International Conference on Robotics and Automation.

[56]  Siddhartha S. Srinivasa,et al.  Manipulation planning on constraint manifolds , 2009, 2009 IEEE International Conference on Robotics and Automation.

[57]  Steven M. LaValle,et al.  Survivability: Measuring and ensuring path diversity , 2009, 2009 IEEE International Conference on Robotics and Automation.

[58]  Shuuji Kajita,et al.  Cybernetic human HRP-4C , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[59]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[60]  Florent Lamiraux,et al.  Whole-body task planning for a humanoid robot: a way to integrate collision avoidance , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[61]  Siddhartha S. Srinivasa,et al.  Addressing pose uncertainty in manipulation planning using Task Space Regions , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[62]  Siddhartha S. Srinivasa,et al.  Probabilistically complete planning with end-effector pose constraints , 2010, 2010 IEEE International Conference on Robotics and Automation.

[63]  Julius Ziegler,et al.  Optimal trajectory generation for dynamic street scenarios in a Frenét Frame , 2010, 2010 IEEE International Conference on Robotics and Automation.

[64]  Matei T. Ciocarlie,et al.  Towards Reliable Grasping and Manipulation in Household Environments , 2010, ISER.

[65]  Geoffrey A. Hollinger,et al.  HERB: a home exploring robotic butler , 2010, Auton. Robots.

[66]  S. Srinivasa,et al.  Push-grasping with dexterous hands: Mechanics and a method , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[67]  Moritz Tenorth,et al.  CRAM — A Cognitive Robot Abstract Machine for everyday manipulation in human environments , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[68]  Jonathan Bohren,et al.  The SMACH High-Level Executive [ROS News] , 2010 .

[69]  Alexander Ferrein,et al.  Design Principles of the Component-Based Robot Software Framework Fawkes , 2010, SIMPAR.

[70]  Steve Cousins,et al.  The SMACH High-Level Executive , 2010 .

[71]  Advait Jain,et al.  EL-E: an assistive mobile manipulator that autonomously fetches objects from flat surfaces , 2010, Auton. Robots.

[72]  Siddhartha S. Srinivasa,et al.  Manipulation Capabilities with Simple Hands , 2010, ISER.

[73]  Siddhartha S. Srinivasa,et al.  Task Space Regions , 2011, Int. J. Robotics Res..

[74]  Siddhartha S. Srinivasa,et al.  Using spatial and temporal contrast for fluent robot-human hand-overs , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[75]  Siddhartha S. Srinivasa,et al.  Human preferences for robot-human hand-over configurations , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[76]  Siddhartha S. Srinivasa,et al.  Addressing cost-space chasms in manipulation planning , 2011, 2011 IEEE International Conference on Robotics and Automation.

[77]  Siddhartha S. Srinivasa,et al.  Manipulation planning with goal sets using constrained trajectory optimization , 2011, 2011 IEEE International Conference on Robotics and Automation.

[78]  Siddhartha S. Srinivasa,et al.  A Framework for Push-Grasping in Clutter , 2011, Robotics: Science and Systems.

[79]  Siddhartha S. Srinivasa,et al.  Learning from Experience in Manipulation Planning: Setting the Right Goals , 2011, ISRR.

[80]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[81]  Siddhartha S. Srinivasa,et al.  Structure discovery in multi-modal data: A region-based approach , 2011, 2011 IEEE International Conference on Robotics and Automation.

[82]  Siddhartha S. Srinivasa,et al.  The MOPED framework: Object recognition and pose estimation for manipulation , 2011, Int. J. Robotics Res..

[83]  Siddhartha S. Srinivasa,et al.  Constrained Manipulation Planning , 2011 .

[84]  W. Marsden.  I and J , 2012 .

[85]  Siddhartha S. Srinivasa,et al.  Autonomous manipulation with a general-purpose simple hand , 2011, Int. J. Robotics Res..