A Novel Gesture Recognition System for Intelligent Interaction with a Nursing-Care Assistant Robot

The expansion of nursing-care assistant robots in smart infrastructure has provided more applications for homecare services, which has raised new demands for smart and natural interaction between humans and robots. This article proposed an innovative hand motion trajectory (HMT) gesture recognition system based on background velocity features. Here, a new wearable wrist-worn camera prototype for gesture’s video collection was designed, and a new method for the segmentation of continuous gestures was shown. Meanwhile, a nursing-care assistant robot prototype was designed for assisting the elderly, which is capable of carrying the elderly with omnidirectional motion and grabbing the specified object at home. In order to evaluate the performance of the gesture recognition system, 10 special gestures were defined as the move commands for interaction with the robot, and 1000 HMT gesture samples were obtained from five subjects for leave-one-subject-out (LOSO) cross-validation classification with an average recognition accuracy of up to 97.34%. Moreover, the performance and practicability of the proposed system were further demonstrated by controlling the omnidirectional movement of the nursing-care assistant robot using the predefined gesture commands.

[1]  Ana-Maria Cretu,et al.  Static and Dynamic Hand Gesture Recognition in Depth Data Using Dynamic Time Warping , 2016, IEEE Transactions on Instrumentation and Measurement.

[2]  Geng Yang,et al.  Finger Angle-Based Hand Gesture Recognition for Smart Infrastructure Using Wearable Wrist-Worn Camera , 2018 .

[3]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[4]  Ruize Xu,et al.  MEMS Accelerometer Based Nonspecific-User Hand Gesture Recognition , 2012, IEEE Sensors Journal.

[5]  Hong Cheng,et al.  An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition , 2016, Pattern Recognit..

[6]  James H. Garrett.  Wearable Computers : An Interface between Humans and Smart Infrastructure Systems , 2002 .

[7]  Mengchao Ma,et al.  A method for correcting non-linear geometric distortion in ultra-wide-angle imaging system , 2013 .

[8]  Luciano Tarricone,et al.  Wearable antennas for applications in remote assistance to elderly people , 2017, 2017 IEEE International Workshop on Measurement and Networking (M&N).

[9]  Mu Zhou,et al.  Latern: Dynamic Continuous Hand Gesture Recognition Using FMCW Radar Sensor , 2018, IEEE Sensors Journal.

[10]  Vijay Sivaraman,et al.  Low-cost flow-based security solutions for smart-home IoT devices , 2016, International Workshop on Ant Colony Optimization and Swarm Intelligence.

[11]  Hongliang Guo,et al.  Structured dynamic time warping for continuous hand trajectory gesture recognition , 2018, Pattern Recognit..

[12]  Naomi Sweller,et al.  Presentation and production: the role of gesture in spatial communication. , 2014, Journal of experimental child psychology.

[13]  Christian Mandel,et al.  Navigating a smart wheelchair with a brain-computer interface interpreting steady-state visual evoked potentials , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Yulei Wang,et al.  Visual tracking and learning using speeded up robust features , 2012, Pattern Recognit. Lett..

[15]  Takanori Shibata,et al.  Psychological and social effects in long-term experiment of robot assisted activity to elderly people at a health service facility for the aged , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[16]  Chih-Yung Chang,et al.  Design and Implementation of an IoT Access Point for Smart Home , 2015 .

[17]  Patrick Olivier,et al.  Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor , 2012, UIST.

[18]  Masatoshi Ishikawa,et al.  High-speed Human / Robot Hand Interaction System , 2015, HRI.

[19]  A. Al-Hamadi,et al.  Gesture Recognition for Alphabets from Hand Motion Trajectory Using Hidden Markov Models , 2007, 2007 IEEE International Symposium on Signal Processing and Information Technology.

[20]  Lihui Wang,et al.  Gesture recognition for human-robot collaboration: A review , 2017, International Journal of Industrial Ergonomics.

[21]  Brigitte Meillon,et al.  Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects , 2011, Personal and Ubiquitous Computing.

[22]  Dan Xu,et al.  Real-time dynamic gesture recognition system based on depth perception for robot navigation , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[23]  Shijie Guo,et al.  Whole-body contact manipulation using tactile information for the nursing-care assistant robot RIBA , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[24]  Thomas Bock,et al.  Towards a Future Robotic Home Environment: A Survey , 2014, Gerontology.

[25]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[26]  Xia Sun,et al.  Similarity Matching-Based Extensible Hand Gesture Recognition , 2015, IEEE Sensors Journal.

[27]  Nick Bryan-Kinns,et al.  Usability and Design of Personal Wearable and Portable Devices for Thermal Comfort in Shared Work Environments , 2016, BCS HCI.

[28]  Partha Pratim Ray.  A survey on Internet of Things architectures , 2018, J. King Saud Univ. Comput. Inf. Sci..

[29]  Kuldip K. Paliwal,et al.  A modification over Sakoe and Chiba's dynamic time warping algorithm for isolated word recognition☆ , 1982 .

[30]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[31]  Sunghyuk Kwon,et al.  Can User-Derived Gesture be Considered as the Best Gesture for a Command?: Focusing on the Commands for Smart Home System , 2012 .

[32]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Olivier Gibaru,et al.  Natural control of an industrial robot using hand gesture recognition with neural networks , 2016, IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society.

[34]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[35]  Mingzhe Jiang,et al.  An IoT-Enabled Stroke Rehabilitation System Based on Smart Wearable Armband and Machine Learning , 2018, IEEE Journal of Translational Engineering in Health and Medicine.

[36]  Toshiharu Mukai,et al.  Development of a nursing-care assistant robot RIBA that can lift a human in its arms , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[37]  Jianping Gou,et al.  A generalized mean distance-based k-nearest neighbor classifier , 2019, Expert Syst. Appl..