Fingertip-based Real Time Tracking and Gesture Recognition for Natural User Interfaces

The widespread deployment of Natural User Interface (NUI) systems in smart terminals such as phones, tablets and TV sets has heightened the need for robust multitouch, speech or facial recognition solutions. Air gestures recognition represents one of the most appealing technologies in the field. This work proposes a fingertip-based approach for hand gesture recognition. The novelty in the proposed system is the tracking principle, where an improved version of the multi-scale mode filtering (MSMF) algorithm is used, and in the classification step, where the proposed set of geometric features provides high discriminative capabilities. Empirically, we conducted an experimental study involving hand gesture recognition – on gestures performed by multiple persons against a variety of backgrounds – in which our approach achieved a global recognition rate of 95.66%.

[1]  G. Roth,et al.  Hand tracking for interactive pattern-based augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[2]  Vasile Gui,et al.  FINGER DETECTION IN VIDEO SEQUENCES USING A NEW SPARSE REPRESENTATION , 2011 .

[3]  Yu Sun,et al.  Static Hand Gesture Recognition and its Application based on Support Vector Machines , 2008, 2008 Ninth ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing.

[4]  Horst Bischof,et al.  Real time appearance based hand tracking , 2008, 2008 19th International Conference on Pattern Recognition.

[5]  Prashan Premaratne,et al.  Consumer electronics control system based on hand gesture moment invariants , 2007 .

[6]  James E. Davis,et al.  Camera-based pointing interface for mobile devices , 2008, 2008 15th IEEE International Conference on Image Processing.

[7]  Le Song,et al.  Real-time 3D Finger Pointing for an Augmented Desk , 2005, AUIC.

[8]  Vasile Gui,et al.  Dynamic hand gesture recognition for human-computer interactions , 2011, 2011 6th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI).

[9]  Barbara Mazzarino,et al.  Finger Tracking Methods Using EyesWeb , 2005, Gesture Workshop.

[10]  Stefan Müller,et al.  Hand Gesture Recognition with a Novel IR Time-of-Flight Range Camera-A Pilot Study , 2007, MIRAGE.

[11]  Jong-Min Kim,et al.  Hand Shape Recognition Using Fingertips , 2008, 2008 Fifth International Conference on Fuzzy Systems and Knowledge Discovery.

[12]  Paulo R. S. Mendonça,et al.  Model-based 3D tracking of an articulated hand , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[13]  Xiaobo Ren,et al.  Fingertip detection and tracking using 2D and 3D information , 2008, 2008 7th World Congress on Intelligent Control and Automation.

[14]  Mubarak Shah,et al.  Recognizing Hand Gestures , 1994, ECCV.

[15]  Jean-Philippe Thiran,et al.  Joint Object Segmentation and Behavior Classification in Image Sequences , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  SeungGwan Lee,et al.  Vision‐Based Finger Action Recognition by Angle Detection and Contour Analysis , 2011 .

[17]  Larry S. Davis,et al.  Real-time foreground-background segmentation using codebook model , 2005, Real Time Imaging.

[18]  Luc Van Gool,et al.  Smart particle filtering for 3D hand tracking , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[19]  Carlo Tomasi,et al.  Efficient Visual Object Tracking with Online Nearest Neighbor Classifier , 2010, ACCV.

[20]  Ying Wu,et al.  Capturing natural hand articulation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[21]  Vasile Gui.  Edge preserving smoothing by multiscale mode filtering , 2008, 2008 16th European Signal Processing Conference.

[22]  Jesús Bescós,et al.  Real-time user independent hand gesture recognition from time-of-flight camera video using static and dynamic models , 2011, Machine Vision and Applications.

[23]  Nikolaos Grammalidis,et al.  A face and gesture recognition system based on an active stereo sensor , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[24]  Joachim Hornegger,et al.  Gesture recognition with a Time-Of-Flight camera , 2008, Int. J. Intell. Syst. Technol. Appl..

[25]  Tobias Höllerer,et al.  Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[26]  Horst Bischof,et al.  PROST: Parallel robust online simple tracking , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Yoichi Sato,et al.  Real-Time Fingertip Tracking and Gesture Recognition , 2002, IEEE Computer Graphics and Applications.

[28]  Maurice Milgram,et al.  A Real Time Fingers Detection by Symmetry Transform Using a Two Cameras System , 2008, ISVC.

[29]  Lars Bretzner,et al.  Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[30]  Björn Stenger,et al.  Model-based hand tracking using a hierarchical Bayesian filter , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Stan Sclaroff,et al.  A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Daeho Lee,et al.  Vision-based remote control system by motion detection and open finger counting , 2009, IEEE Transactions on Consumer Electronics.

[33]  G. Simion,et al.  A ToF 3D database for hand gesture recognition , 2012, 2012 10th International Symposium on Electronics and Telecommunications.

[34]  R. Herpers,et al.  Fast hand gesture recognition for real-time teleconferencing applications , 2001, Proceedings IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems.

[35]  Margrit Betke,et al.  Counting Fingers in Real Time: A Webcam-Based Human-Computer Interface with Game Applications , 2003, HCI.