Human–Robot Interaction by Understanding Upper Body Gestures

In this paper, a human–robot interaction system based on a novel combination of sensors is proposed. It allows one person to interact with a humanoid social robot using natural body language. The robot understands the meaning of human upper body gestures and expresses itself by using a combination of body movements, facial expressions, and verbal language. A set of 12 upper body gestures is involved for communication. This set also includes gestures with human–object interactions. The gestures are characterized by head, arm, and hand posture information. The wearable Immersion CyberGlove II is employed to capture the hand posture. This information is combined with the head and arm posture captured from Microsoft Kinect. This is a new sensor solution for human-gesture capture. Based on the posture data from the CyberGlove II and Kinect, an effective and real-time human gesture recognition method is proposed. The gesture understanding approach based on an innovative combination of sensors is the main contribution of this paper. To verify the effectiveness of the proposed gesture recognition method, a human body gesture data set is built. The experimental results demonstrate that our approach can recognize the upper body gestures with high accuracy in real time. In addition, for robot motion generation and control, a novel online motion planning method is proposed. In order to generate appropriate dynamic motion, a quadratic programming (QP)-based dual-arms kinematic motion generation scheme is proposed, and a simplified recurrent neural network is employed to solve the QP problem. The integration of a handshake within the HRI system illustrates the effectiveness of the proposed online generation method.

[1]  Yoichiro Maeda,et al.  Body mapping from human demonstrator to inverted-pendulum mobile robot for learning from observation , 2012, 2012 IEEE International Conference on Fuzzy Systems.

[2]  Fan-Tien Cheng,et al.  Resolving manipulator redundancy under inequality constraints , 1994, IEEE Trans. Robotics Autom..

[3]  Lola Cannery,et al.  I Show You how I Like You-Can You Read it in My Face , 2001 .

[4]  Alexander H. Waibel,et al.  Natural human-robot interaction using speech, head pose and gestures , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[5]  Jianxin Wu,et al.  mCENTRIST: A Multi-Channel Feature Generation Mechanism for Scene Categorization , 2014, IEEE Transactions on Image Processing.

[6]  H. Teleb,et al.  Data glove integration with 3D virtual environments , 2012, 2012 International Conference on Systems and Informatics (ICSAI2012).

[7]  Yunong Zhang,et al.  Acceleration-level repetitive motion planning of redundant planar robots solved by a simplified LVI-based primal-dual neural network , 2013 .

[8]  Paulo Menezes,et al.  Face tracking and hand gesture recognition for human-robot interaction , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[9]  Rajiv V. Dubey,et al.  A weighted least-norm solution based scheme for avoiding joint limits for redundant joint manipulators , 1993, IEEE Trans. Robotics Autom..

[10]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[11]  Lola Cañamero,et al.  I show you how I like you - can you read it in my face? [robotics] , 2001, IEEE Trans. Syst. Man Cybern. Part A.

[12]  Meinard Müller,et al.  Motion templates for automatic classification and retrieval of motion capture data , 2006, SCA '06.

[13]  Fabio Tesser,et al.  Interpretation of Emotional Body Language Displayed by a Humanoid Robot: A Case Study with Children , 2013, Int. J. Soc. Robotics.

[14]  Aryel Beck,et al.  Using Perlin Noise to Generate Emotional Expressions in a Robot , 2013, CogSci.

[15]  Binghuang Cai,et al.  Different-Level Redundancy-Resolution and Its Equivalent Relationship Analysis for Robot Manipulators Using Gradient-Descent and Zhang 's Neural-Dynamic Methods , 2012, IEEE Transactions on Industrial Electronics.

[16]  Yunong Zhang,et al.  Design and experimentation of acceleration-level drift-free scheme aided by two recurrent neural networks , 2013 .

[17]  Michael A. Goodrich,et al.  Human-Robot Interaction: A Survey , 2008, Found. Trends Hum. Comput. Interact..

[18]  Yunong Zhang,et al.  Equivalence of Different-level Schemes for Repetitive Motion Planning of Redundant Robots , 2013 .

[19]  Brett Stevens,et al.  Emotional body language displayed by artificial agents , 2012, TIIS.

[20]  Nicole C. Krämer,et al.  Effects of Embodied Interface Agents and Their Gestural Activity , 2003, IVA.

[21]  Hamid D. Taghirad,et al.  Kinematic Analysis of a Macro–Micro Redundantly Actuated Parallel Manipulator , 2008, Adv. Robotics.

[22]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[23]  Zhiguo Cao,et al.  Entropic image thresholding based on GLGM histogram , 2014, Pattern Recognit. Lett..

[24]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[25]  Georgios Pierris,et al.  An interactive tool for designing complex robot motion patterns , 2009, 2009 IEEE International Conference on Robotics and Automation.

[26]  Shugen Ma,et al.  Time-optimal control of kinematically redundant manipulators with limit heat characteristics of actuators , 2002, Adv. Robotics.

[27]  Hamid D. Taghirad,et al.  Dynamic Analysis of a Macro–Micro Redundantly Actuated Parallel Manipulator , 2008, Adv. Robotics.

[28]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[29]  Linda B. Smith,et al.  The dynamic lift of developmental process. , 2007, Developmental science.

[30]  Yann LeCun,et al.  Learning a similarity metric discriminatively, with application to face verification , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[31]  Kerstin Dautenhahn,et al.  Socially intelligent robots: dimensions of human–robot interaction , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[32]  Constantine D. Spyropoulos,et al.  HUMAN-ROBOT INTERACTION BASED ON SPOKEN NATURAL LANGUAGE DIALOGUE , 2001 .

[33]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[34]  Lik-Kwan Shark,et al.  Immersive manipulation of virtual objects through glove-based hand gesture interaction , 2011, Virtual Reality.

[35]  J. Cassell,et al.  Nudge nudge wink wink: elements of face-to-face conversation for embodied conversational agents , 2001 .

[36]  Pierre-Brice Wieber,et al.  Kinematic Control of Redundant Manipulators: Generalizing the Task-Priority Framework to Inequality Task , 2011, IEEE Transactions on Robotics.

[37]  Magdalena D. Bugajska,et al.  Building a Multimodal Human-Robot Interface , 2001, IEEE Intell. Syst..

[38]  Zhi-Jun Zhang,et al.  Equivalence of Different-level Schemes for Repetitive Motion Planning of Redundant Robots: Equivalence of Different-level Schemes for Repetitive Motion Planning of Redundant Robots , 2014 .

[39]  Kalpana C. Jondhale,et al.  Shape matching and object recognition using shape contexts , 2010 .

[40]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[41]  Rainer Stiefelhagen,et al.  Visual recognition of pointing gestures for human-robot interaction , 2007, Image Vis. Comput..

[42]  Hiroshi Ishiguro,et al.  Human-like natural behavior generation based on involuntary motions for humanoid robots , 2004, Robotics Auton. Syst..

[43]  J. Osborne,et al.  Sample size and subject to item ratio in principal components analysis. , 2004 .

[44]  A.M. Dias,et al.  Comments on Manipulability Measure in Redundant Planar Manipulators , 2006, 2006 IEEE 3rd Latin American Robotics Symposium.

[45]  Yunong Zhang,et al.  Variable Joint-Velocity Limits of Redundant Robot Manipulators Handled by Quadratic Programming , 2013, IEEE/ASME Transactions on Mechatronics.

[46]  Zhijun Zhang,et al.  Acceleration-Level Cyclic-Motion Generation of Constrained Redundant Robots Tracking Different Paths , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[47]  Nicoletta Adamo-Villani,et al.  Two gesture recognition systems for immersive math education of the deaf , 2007, IMMERSCOM.

[48]  Sven Behnke,et al.  The humanoid museum tour guide Robotinho , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[49]  Sebastian Thrun,et al.  A Gesture Based Interface for Human-Robot Interaction , 2000, Auton. Robots.

[50]  Dongsheng Guo,et al.  A New Inequality-Based Obstacle-Avoidance MVN Scheme and Its Application to Redundant Robot Manipulators , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[51]  Helman Stern,et al.  Sensors for Gesture Recognition Systems , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[52]  Ke Chen,et al.  A simplified LVI-based primal-dual neural network for repetitive motion planning of PA10 robot manipulator starting from different initial states , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[53]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[54]  Alexander H. Waibel,et al.  Enabling Multimodal Human–Robot Interaction for the Karlsruhe Humanoid Robot , 2007, IEEE Transactions on Robotics.

[55]  Ying Wu,et al.  Mining actionlet ensemble for action recognition with depth cameras , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Daniel Thalmann,et al.  Human-virtual human interaction by upper body gesture understanding , 2013, VRST '13.

[57]  Yangmin Li,et al.  Inverse kinematics analysis for the arm of a mobile humanoid robot based on the closed-loop algorithm , 2009, 2009 International Conference on Information and Automation.