Comparison of Interaction Modalities for Mobile Indoor Robot Guidance: Direct Physical Interaction, Person Following, and Pointing Control

Three advanced natural interaction modalities for mobile robot guidance in an indoor environment were developed and compared using two tasks and quantitative metrics to measure performance and workload. The first interaction modality is based on direct physical interaction requiring the human user to push the robot in order to displace it. The second and third interaction modalities exploit a 3-D vision-based human-skeleton tracking allowing the user to guide the robot by either walking in front of it or by pointing toward a desired location. In the first task, the participants were asked to guide the robot between different rooms in a simulated physical apartment requiring rough movement of the robot through designated areas. The second task evaluated robot guidance in the same environment through a set of waypoints, which required accurate movements. The three interaction modalities were implemented on a generic differential drive mobile platform equipped with a pan-tilt system and a Kinect camera. Task completion time and accuracy were used as metrics to assess the users' performance, while the NASA-TLX questionnaire was used to evaluate the users' workload. A study with 24 participants indicated that choice of interaction modality had significant effect on completion time (F (2, 61) = 84.874, p <; 0.001), accuracy (F (2, 29) = 4.937, p = 0.016), and workload (F (2, 68) = 11.948, p <; 0.001). The direct physical interaction required less time, provided more accuracy and less workload than the two contactless interaction modalities. Between the two contactless interaction modalities, the person-following interaction modality was systematically better than the pointing-control one: The participants completed the tasks faster with less workload.

[1]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[2]  José María Martínez-Otzeta,et al.  People Following Behaviour in an Industrial Enviroment Using Laser and Stereo Camera , 2010, IEA/AIE.

[3]  Roland Siegwart,et al.  Multiclass Multimodal Detection and Tracking in Urban Environments * , 2009, FSR.

[4]  Shin'ichi Yuta,et al.  Operation direction to a mobile robot by projection lights , 2005, IEEE Workshop on Advanced Robotics and its Social Impacts, 2005..

[5]  Terrence Fong,et al.  Novel interfaces for remote driving: gesture, haptic, and PDA , 2001, SPIE Optics East.

[6]  Tiffany L. Chen,et al.  Lead me by the hand: Evaluation of a direct physical interface for nursing assistant robots , 2010, 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[7]  Gaurav S. Sukhatme,et al.  People tracking and following with mobile robot using an omnidirectional camera and a laser , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[8]  Joshua G. Hale,et al.  "Sticky Hands": learning and generalization for cooperative physical interactions with a humanoid robot , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[9]  Ankit Chaudhary,et al.  Intelligent Approaches to interact with Machines using Hand Gesture Recognition in Natural way: A Survey , 2011, ArXiv.

[10]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[11]  Robin R. Murphy,et al.  Hand gesture recognition with depth images: A review , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[12]  Tiffany L. Chen,et al.  A Direct Physical Interface for Navigation and Positioning of a Robotic Nursing Assistant , 2011, Adv. Robotics.

[13]  Zhe Xu,et al.  A point-and-click interface for the real world: Laser designation of objects for mobile manipulation , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[14]  Loreto Susperregi,et al.  Laser Based People Following Behaviour in an Emergency Environment , 2009, ICIRA.

[15]  Alex Pentland,et al.  Real-time self-calibrating stereo person tracking using 3-D shape estimation from blob features , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[16]  Gaurav S. Sukhatme,et al.  Haptic Teleoperation of a Mobile Robot: A User Study , 2005, Presence: Teleoperators & Virtual Environments.

[17]  Trevor Darrell,et al.  Integrated Person Tracking Using Stereo, Color, and Pattern Detection , 2000, International Journal of Computer Vision.

[18]  Guillaume Doisy,et al.  Position-invariant, real-time gesture recognition based on dynamic time warping , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[19]  Xose Manuel Pardo,et al.  Feature analysis for human recognition and discrimination: Application to a person-following behaviour in a mobile robot , 2012, Robotics Auton. Syst..

[20]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[21]  Alessandro De Luca,et al.  Collision Detection and Safe Reaction with the DLR-III Lightweight Manipulator Arm , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[22]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[23]  S. R. Searle,et al.  Generalized, Linear, and Mixed Models , 2005 .

[24]  Takayuki Kanda,et al.  Changes in Usage of an Indoor Public Space: Analysis of One Year of Person Tracking , 2015, IEEE Transactions on Human-Machine Systems.

[25]  Jose Antonio Diego-Mas,et al.  Using Kinect™ sensor in observational methods for assessing postures at work. , 2014, Applied ergonomics.

[26]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[27]  Mehrdad R. Kermani,et al.  Applications of Artificial Intelligence in Safe Human–Robot Interactions , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[29]  François Michaud,et al.  Taking your robot for a walk: Force-guiding a mobile robot using compliant arms , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[30]  Maya Cakmak,et al.  Robot Programming by Demonstration with situated spatial language understanding , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Oliver Kroemer,et al.  Learning to select and generalize striking movements in robot table tennis , 2012, AAAI Fall Symposium: Robots Learning Interactively from Human Teachers.

[32]  Saso Koceski,et al.  Design and Evaluation of Cell Phone Pointing Interface for Robot Control , 2012 .

[33]  Advait Jain,et al.  Pulling open novel doors and drawers with equilibrium point control , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[34]  Nikitas M. Sgouros,et al.  Integrating WAP-based wireless devices in robot teleoperation environments , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[35]  Nassir Navab,et al.  Human skeleton tracking from depth data using geodesic distances and optical flow , 2012, Image Vis. Comput..

[36]  Hiroshi Mizoguchi,et al.  Development of a Person Following Robot with Vision Based Target Detection , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[37]  Helge J. Ritter,et al.  Multi-modal human-machine communication for instructing robot grasping tasks , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[38]  Jörg Stückler,et al.  Learning to interpret pointing gestures with a time-of-flight camera , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[39]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[40]  Nasser Kehtarnavaz,et al.  Improving Human Action Recognition Using Fusion of Depth Camera and Inertial Sensors , 2015, IEEE Transactions on Human-Machine Systems.

[41]  Neville Hogan,et al.  Impedance Control: An Approach to Manipulation , 1984, 1984 American Control Conference.

[42]  Yael Edan,et al.  Adaptive Person-Following Algorithm Based on Depth Images and Mapping * , 2012 .

[43]  Alessandro De Luca,et al.  Collision detection and reaction: A contribution to safe physical Human-Robot Interaction , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[44]  Guillaume Doisy Sensorless collision detection and control by physical interaction for wheeled mobile robots , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[45]  Javaid Iqbal,et al.  On the Improvement of Multi-Legged Locomotion over Difficult Terrains Using a Balance Stabilization Method: , 2012 .

[46]  Wolfram Burgard,et al.  An evaluation of the RGB-D SLAM system , 2012, 2012 IEEE International Conference on Robotics and Automation.

[47]  Xia Liu,et al.  Hand gesture recognition using depth data , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[48]  Panagiotis K. Artemiadis,et al.  On the role duality and switching in human-robot cooperation: An adaptive approach , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[49]  Alessandro De Luca,et al.  An adapt-and-detect actuator FDI scheme for robot manipulators , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[50]  Jun Miura,et al.  Stereo-Based Multi-person Tracking Using Overlapping Silhouette Templates , 2010, 2010 20th International Conference on Pattern Recognition.

[51]  Guillaume Doisy,et al.  Spatially unconstrained, gesture-based human-robot interaction , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[52]  Terrence Fong,et al.  A Safeguarded Teleoperation Controller , 2001 .

[53]  Andrew Wilson,et al.  Pointing in Intelligent Environments with the WorldCursor , 2003, INTERACT.

[54]  Huosheng Hu,et al.  Multisensor-Based Human Detection and Tracking for Mobile Service Robots , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[55]  Kai Oliver Arras,et al.  Place-dependent people tracking , 2011, Int. J. Robotics Res..

[56]  Wayne J. Book,et al.  Force reflecting teleoperation with adaptive impedance control , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[57]  Roberto Cipolla,et al.  Uncalibrated Stereo Vision with Pointing for a Man-Machine Interface , 1994, MVA.

[58]  Luis Salgado,et al.  Efficient spatio-temporal hole filling strategy for Kinect depth maps , 2012, Electronic Imaging.

[59]  Pierre-Yves Oudeyer,et al.  The Impact of Human–Robot Interfaces on the Learning of Visual Objects , 2013, IEEE Transactions on Robotics.

[60]  William D. Smart,et al.  Evaluation of Interfaces for 3d Pointing , 2015, HRI.

[61]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004 .

[62]  Manuela M. Veloso,et al.  Effective Multi-Model Motion Tracking using Action Models , 2009, Int. J. Robotics Res..

[63]  S. R. Searle,et al.  Generalized, Linear, and Mixed Models: McCulloch/Generalized, Linear, and Mixed Models , 2005 .

[64]  Donald A. Norman,et al.  Natural user interfaces are not natural , 2010, INTR.

[65]  Yael Edan,et al.  Advanced methods for displays and remote control of robots. , 2011, Applied ergonomics.

[66]  Wolfram Burgard,et al.  People Tracking with Mobile Robots Using Sample-Based Joint Probabilistic Data Association Filters , 2003, Int. J. Robotics Res..

[67]  Charles E. Thorpe,et al.  Collaborative control: a robot-centric model for vehicle teleoperation , 2001 .

[68]  Satoru Shibata,et al.  Human-robot interface with instruction of neck movement using laser pointer , 2011, 2011 IEEE/SICE International Symposium on System Integration (SII).

[69]  Kazuhiro Kosuge,et al.  Dance Step Estimation Method Based on HMM for Dance Partner Robot , 2007, IEEE Transactions on Industrial Electronics.

[70]  Hikaru Inooka,et al.  Variable impedance control of a robot for cooperation with a human , 1995, Proceedings of 1995 IEEE International Conference on Robotics and Automation.

[71]  Steven A. Shafer,et al.  XWand: UI for intelligent spaces , 2003, CHI '03.

[72]  Alessandro De Luca,et al.  Sensorless Robot Collision Detection and Hybrid Force/Motion Control , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[73]  Daniele Pucci,et al.  Nonlinear control of unicycle-like robots for person following , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[74]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  Charles C. Kemp,et al.  Human-Robot Interaction for Cooperative Manipulation: Handing Objects to One Another , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[76]  Fazel Naghdy,et al.  Application of Adaptive Controllers in Teleoperation Systems: A Survey , 2014, IEEE Transactions on Human-Machine Systems.

[77]  John Kenneth Salisbury,et al.  Towards a personal robotics development platform: Rationale and design of an intrinsically safe personal robot , 2008, 2008 IEEE International Conference on Robotics and Automation.

[78]  Christian Laugier,et al.  On leader following and classification , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[79]  Wendy A. Rogers,et al.  “Commanding Your Robot” Older Adults’ Preferences for Methods of Robot Control , 2012, Proceedings of the Human Factors and Ergonomics Society ... Annual Meeting. Human Factors and Ergonomics Society. Annual Meeting.

[80]  Rainer Stiefelhagen,et al.  Visual recognition of pointing gestures for human-robot interaction , 2007, Image Vis. Comput..

[81]  Nebojsa Jojic,et al.  Detection and estimation of pointing gestures in dense disparity maps , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[82]  Maria Pateraki,et al.  Visual estimation of pointed targets for robot guidance via fusion of face pose and hand orientation , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[83]  Polychronis Kondaxakis,et al.  Real-time recognition of pointing gestures for robot to robot interaction , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[84]  Alin Albu-Schäffer,et al.  Soft-tissue injury in robotics , 2010, 2010 IEEE International Conference on Robotics and Automation.

[85]  Yael Edan,et al.  Human-Robot Interaction through 3D Vision and Force Control , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[86]  Florian Schmidt,et al.  Rollin' Justin - Design considerations and realization of a mobile platform for a humanoid upper body , 2009, 2009 IEEE International Conference on Robotics and Automation.

[87]  Subhash Challa,et al.  Vision Based Target Tracking for Autonomous Land Vehicle Navigation: A Brief Survey , 2009 .

[88]  Roberto Cipolla,et al.  Human-robot interface by pointing with uncalibrated stereo vision , 1996, Image Vis. Comput..

[89]  Michael A. Goodrich,et al.  Ecological Interfaces for Improving Mobile Robot Teleoperation , 2007, IEEE Transactions on Robotics.

[90]  Yuichi Motai,et al.  Human tracking from a mobile agent: Optical flow and Kalman filter arbitration , 2012, Signal Process. Image Commun..

[91]  Ninad Pradhan,et al.  Mobile Robot Navigation for Person Following in Indoor Environments , 2013 .

[92]  Luca Iocchi,et al.  Person following through appearance models and stereo vision using a mobile robot , 2007, VISAPP.

[93]  Bruno Lara,et al.  Learning the Relation of Motion Control and Gestures Through Self-Exploration , 2012 .