Towards Zero Re-Training for Long-Term Hand Gesture Recognition via Ultrasound Sensing

While myoelectric pattern recognition is a prevailing way for gesture recognition, the inherent nonstationarity of electromyography signals hinders its long-term application. This study aims to prove a hypothesis that morphological information of muscle contraction detected by ultrasound image is potentially suitable for long-term use. A set of ultrasound-based algorithms are proposed to realize robust hand gesture recognition over multiple days, with user training only at the first day. A markerless calibration algorithm is first presented to position the ultrasound probe during donning and doffing; an algorithm combining speeded-up robust features and bag-of-features model being immune to ultrasound probe shift and rotation is then introduced; a self-enhancing classification method is next adopted to update classification model automatically by incorporating useful knowledge from testing data; finally the performance of long-term hand gesture recognition with zero re-training is validated by a six-day experiment of six healthy subjects, whose outcomes strongly support the hypothesis with about 94% of gesture recognition accuracy for each testing day. This study confirms the feasibility of adoption of ultrasound sensing for long-term musculature related applications.

[1]  Nicolas D. Georganas,et al.  Real-Time Hand Gesture Detection and Recognition Using Bag-of-Features and Support Vector Machine Techniques , 2011, IEEE Transactions on Instrumentation and Measurement.

[2]  Chong-Wah Ngo,et al.  Towards optimal bag-of-features for object categorization and semantic video retrieval , 2007, CIVR '07.

[3]  Blair A. Lock,et al.  Adaptive Pattern Recognition of Myoelectric Signals: Exploration of Conceptual Framework and Practical Algorithms , 2009, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[4]  Xinjun Sheng,et al.  Cascaded Adaptation Framework for Fast Calibration of Myoelectric Control , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[5]  Honghai Liu,et al.  Towards Wearable A-Mode Ultrasound Sensing for Real-Time Finger Motion Recognition , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[6]  Honghai Liu,et al.  Ultrasound-Based Sensing Models for Finger Motion Classification , 2018, IEEE Journal of Biomedical and Health Informatics.

[7]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[9]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[10]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Jana Kosecka,et al.  Real-Time Classification of Hand Motions Using Ultrasound Imaging of Forearm Muscles , 2016, IEEE Transactions on Biomedical Engineering.

[12]  Claudio Castellini,et al.  Ultrasound image features of the wrist are linearly related to finger positions , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Dingguo Zhang,et al.  Application of a self-enhancing classification method to electromyography pattern recognition for multifunctional prosthesis control , 2013, Journal of NeuroEngineering and Rehabilitation.

[14]  Jing-Yi Guo,et al.  Recognition of Finger Flexion from Ultrasound Image with Optical Flow: A Preliminary Study , 2010, 2010 International Conference on Biomedical Engineering and Computer Science.

[15]  Mike Fraser,et al.  EchoFlex: Hand Gesture Recognition using Ultrasound Imaging , 2017, CHI.

[16]  C. Castellini,et al.  Using Ultrasound Images of the Forearm to Predict Finger Positions , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[17]  Hong-Bo Xie,et al.  Estimation of wrist angle from sonomyography using support vector machine and artificial neural network models. , 2009, Medical engineering & physics.

[18]  Xiao Liu,et al.  Stacked deep polynomial network based representation learning for tumor classification with small ultrasound image dataset , 2016, Neurocomputing.

[19]  Xinjun Sheng,et al.  Towards Zero Retraining for Myoelectric Control Based on Common Model Component Analysis , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[20]  Claudio Castellini,et al.  A realistic implementation of ultrasound imaging as a human-machine interface for upper-limb amputees , 2013, Front. Neurorobot..

[21]  Kevin B. Englehart,et al.  A robust, real-time control scheme for multifunction myoelectric control , 2003, IEEE Transactions on Biomedical Engineering.

[22]  Stefano Stramigioli,et al.  Myoelectric forearm prostheses: state of the art from a user-centered perspective. , 2011, Journal of rehabilitation research and development.

[23]  T. Van de Cruys,et al.  A Comparison of Bag of Words and Syntax-based Approaches for Word Categorization , 2008 .

[24]  Jing-Yi Guo,et al.  Recognition of finger flexion motion from ultrasound image: a feasibility study. , 2012, Ultrasound in medicine & biology.

[25]  Huzefa Rangwala,et al.  Novel Method for Predicting Dexterous Individual Finger Movements by Imaging Muscle Activity Using a Wearable Ultrasonic System , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[26]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[27]  Frédéric Jurie,et al.  Creating efficient codebooks for visual recognition , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[28]  Ryutarou Ohbuchi,et al.  Dense sampling and fast encoding for 3D model retrieval using bag-of-visual features , 2009, CIVR '09.

[29]  Xinjun Sheng,et al.  User adaptation in long-term, open-loop myoelectric training: implications for EMG pattern recognition in prosthesis control , 2015, Journal of neural engineering.

[30]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[31]  Xinjun Sheng,et al.  Reduced Daily Recalibration of Myoelectric Prosthesis Classifiers Based on Domain Adaptation , 2016, IEEE Journal of Biomedical and Health Informatics.

[32]  Frédéric Jurie,et al.  Sampling Strategies for Bag-of-Features Image Classification , 2006, ECCV.

[33]  Claudio Castellini,et al.  A Comparative Analysis of Three Non-Invasive Human-Machine Interfaces for the Disabled , 2014, Front. Neurorobot..

[34]  Claudio Castellini,et al.  Ultrasound imaging as a human-machine interface in a realistic scenario , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[35]  Erik Scheme,et al.  Electromyogram pattern recognition for control of powered upper-limb prostheses: state of the art and challenges for clinical use. , 2011, Journal of rehabilitation research and development.

[36]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[37]  Robert Marti,et al.  Which is the best way to organize/classify images by content? , 2007, Image Vis. Comput..

[38]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[39]  Huosheng Hu,et al.  Support Vector Machine-Based Classification Scheme for Myoelectric Control Applied to Upper Limb , 2008, IEEE Transactions on Biomedical Engineering.