Recognition of Fingerspelling Sequences in Polish Sign Language Using Point Clouds Obtained from Depth Images

The paper presents a method for recognizing sequences of static letters of the Polish finger alphabet using the point cloud descriptors: viewpoint feature histogram, eigenvalues-based descriptors, ensemble of shape functions, and global radius-based surface descriptor. Each sequence is understood as quick highly coarticulated motions, and the classification is performed by networks of hidden Markov models trained by transitions between postures corresponding to particular letters. Three kinds of the left-to-right Markov models of the transitions, two networks of the transition models—independent and dependent on a dictionary—as well as various combinations of point cloud descriptors are examined on a publicly available dataset of 4200 executions (registered as depth map sequences) prepared by the authors. The hand shape representation proposed in our method can also be applied for recognition of hand postures in single frames. We confirmed this using a known, challenging American finger alphabet dataset with about 60,000 depth images.

[1]  Marian Wysocki,et al.  Recognition of Hand Posture Based on a Point Cloud Descriptor and a Feature of Extended Fingers , 2016, J. Autom. Mob. Robotics Intell. Syst..

[2]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[3]  Mariusz Oszust,et al.  Recognition of Hand Gestures Observed by Depth Cameras , 2015 .

[4]  Zoltan-Csaba Marton,et al.  Hierarchical object geometric categorization and appearance classification for mobile manipulation , 2010, 2010 10th IEEE-RAS International Conference on Humanoid Robots.

[5]  Yong Hu.  Finger spelling recognition using depth information and support vector machine , 2018, Multimedia Tools and Applications.

[6]  Vladimir Naumovich Vapni.  The Nature of Statistical Learning Theory , 1995 .

[7]  Dawid Warchol.  Hand posture recognition using modified Ensemble of Shape Functions and Global Radius-based Surface Descriptor , 2018, Comput. Sci..

[8]  Markus Vincze,et al.  Ensemble of shape functions for 3D object classification , 2011, 2011 IEEE International Conference on Robotics and Biomimetics.

[9]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[10]  Akira Iwata,et al.  Hand alphabet recognition using morphological PCA and neural networks , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[11]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[12]  Alex Zelinsky,et al.  Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf] , 2009, IEEE Robotics & Automation Magazine.

[13]  Riccardo Leonardi,et al.  XKin: an open source framework for hand pose and gesture recognition using kinect , 2014, The Visual Computer.

[14]  Guillermo Cámara Chávez,et al.  Finger Spelling Recognition from RGB-D Information Using Kernel Descriptor , 2013, SIBGRAPI.

[15]  Xiaodong Yang,et al.  Histogram of 3D Facets: A characteristic descriptor for hand gesture recognition , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[16]  Gary R. Bradski,et al.  Fast 3D recognition and pose using the Viewpoint Feature Histogram , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Yue Wang,et al.  Real-time hand posture recognition based on hand dominant line using kinect , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[18]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[19]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[20]  Woei-Chyn Chu,et al.  Hand gesture recognition for post-stroke rehabilitation using leap motion , 2017, 2017 International Conference on Applied System Innovation (ICASI).

[21]  Bodo Rosenhahn,et al.  Real-Time Sign Language Recognition Using a Consumer Depth Camera , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[22]  Carlo Tomasi,et al.  Fingerspelling Recognition through Classification of Letter-to-Letter Transitions , 2009, ACCV.

[23]  Karl-Friedrich Kraiss.  Advanced Man-Machine Interaction: Fundamentals and Implementation (Signals and Communication Technology) , 2006 .

[24]  Chong Wang,et al.  Superpixel-Based Hand Gesture Recognition With Kinect Depth Camera , 2015, IEEE Transactions on Multimedia.

[25]  Lale Akarun,et al.  Real time hand pose estimation using depth sensors , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[26]  Eun-Jung Holden,et al.  Dynamic Fingerspelling Recognition using Geometric and Motion Features , 2006, 2006 International Conference on Image Processing.

[27]  Edwin Jonathan Escobedo Cardenas,et al.  Finger Spelling Recognition from Depth Data Using Direction Cosines and Histogram of Cumulative Magnitudes , 2015, 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images.

[28]  Gregory Shakhnarovich,et al.  Lexicon-free fingerspelling recognition from video: Data, models, and signer adaptation , 2017, Comput. Speech Lang..

[29]  Nico Blodow,et al.  Combined 2D–3D categorization and classification for multimodal perception systems , 2011, Int. J. Robotics Res..

[30]  Gregory Shakhnarovich,et al.  American sign language fingerspelling recognition with phonological feature-based tandem models , 2012, 2012 IEEE Spoken Language Technology Workshop (SLT).

[31]  Hao Wang,et al.  Depth-Projection-Map-Based Bag of Contour Fragments for Robust Hand Gesture Recognition , 2017, IEEE Transactions on Human-Machine Systems.

[32]  Alex Waibel,et al.  Readings in speech recognition , 1990 .

[33]  Ming C. Leu,et al.  Recognition of Finger Spelling of American Sign Language with Artificial Neural Network Using Position/Orientation Sensors and Data Glove , 2005, ISNN.

[34]  Cai-Zhi Yang.  Static Gesture Recognition Algorithm Based on Upper Triangular Image Texture and Recursive Graph , 2017 .

[35]  Stefan Hinz,et al.  Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers , 2015 .

[36]  Stephan Liwicki,et al.  Automatic recognition of fingerspelled words in British Sign Language , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.