User-independent system for sign language finger spelling recognition

Abstract We propose in this paper a framework for recognizing the sign language alphabet. To separate hand images from complex backgrounds, we use skin colour and texture attributes with neural networks. The recognition process is based on the combination of three shape descriptors: Discrete orthogonal Tchebichef moments applied on both internal and external outlines hand, Hu moments and a set of geometric features derived from the convex hull that encloses the hand shape taking into account the hand orientation. The recognition is carried out using KNN and SVM classifiers. The proposed descriptors are combined in several sequential and parallel manners and applied on different datasets. The obtained results are compared to existing works.

[1]  Paul F. Whelan,et al.  Image segmentation based on the integration of colour-texture descriptors - A review , 2011, Pattern Recognit..

[2]  Surendra Ranganath,et al.  Real-time gesture recognition system and application , 2002, Image Vis. Comput..

[3]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[4]  Malin Premaratne,et al.  Hand gesture tracking and recognition system using Lucas-Kanade algorithms for control of consumer electronics , 2013, Neurocomputing.

[5]  Saliha Aouat,et al.  Comparison of silhouettes descriptions , 2005 .

[6]  Saliha Aouat,et al.  Matching Descriptors of Noisy Outline Shapes , 2010, Int. J. Image Graph..

[7]  Alex Pentland,et al.  Real-time American Sign Language recognition from video using hidden Markov models , 1995 .

[8]  Chih-Lyang Hwang,et al.  The command control by hand gesture with Hu and contour sequence moments and probability neural network , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[9]  Saliha Aouat,et al.  Shape from Shading with and without Boundary Conditions , 2014 .

[10]  Nadia Baha,et al.  FPGA implementation for stereo matching algorithm , 2013, 2013 Science and Information Conference.

[11]  Ronald L. Graham,et al.  An Efficient Algorithm for Determining the Convex Hull of a Finite Planar Set , 1972, Inf. Process. Lett..

[12]  Richard Bowden,et al.  Sign Language Recognition , 2011, Visual Analysis of Humans.

[13]  Khaled Assaleh,et al.  Recognition of Arabic Sign Language Alphabet Using Polynomial Classifiers , 2005, EURASIP J. Adv. Signal Process..

[14]  David Windridge,et al.  A Linguistic Feature Vector for the Visual Interpretation of Sign Language , 2004, ECCV.

[15]  Saliha Aouat,et al.  Solving the perspective Shape from Shading problem using a new integration method , 2013, 2013 Science and Information Conference.

[16]  Saliha Aouat,et al.  La reconnaissance des silhouettes d’objets en utilisant une description textuelle , 2008, 2008 Canadian Conference on Electrical and Computer Engineering.

[17]  Abraham Kandel,et al.  A Feature-Based Serial Approach to Classifier Combination , 2002, Pattern Analysis and Applications.

[18]  Khaled Assaleh,et al.  Video-based signer-independent Arabic sign language recognition using hidden Markov models , 2009, Appl. Soft Comput..

[19]  Omar M. Al-Jarrah,et al.  Recognition of gestures in Arabic sign language using neuro-fuzzy systems , 2001, Artif. Intell..

[20]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.

[21]  Z. Zenn Bien,et al.  A dynamic gesture recognition system for the Korean sign language (KSL) , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[22]  Andrew W. Fitzgibbon,et al.  Direct Least Square Fitting of Ellipses , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Saliha Aouat,et al.  3D object indexing and recognition , 2008, Appl. Math. Comput..

[24]  Wen Gao,et al.  A Chinese sign language recognition system based on SOFM/SRN/HMM , 2004, Pattern Recognit..

[25]  Hermann Ney,et al.  Spoken language processing techniques for sign language recognition and translation , 2008, Technology and Disability.

[26]  Jochen Triesch,et al.  Classification of hand postures against complex backgrounds using elastic graph matching , 2002, Image Vis. Comput..

[27]  Nicolas D. Georganas,et al.  Real-Time Hand Gesture Detection and Recognition Using Bag-of-Features and Support Vector Machine Techniques , 2011, IEEE Transactions on Instrumentation and Measurement.

[28]  Jianbo Su,et al.  Natural hand posture recognition based on Zernike moments and hierarchical classifier , 2008, 2008 IEEE International Conference on Robotics and Automation.

[29]  Hermann Ney,et al.  Geometric Features for Improving Continuous Appearance-based Sign Language Recognition , 2006, BMVC.

[30]  Daniel Kelly,et al.  A person independent system for recognition of hand postures used in sign language , 2010, Pattern Recognit. Lett..

[31]  Nikolaos G. Bourbakis,et al.  A survey of skin-color modeling and detection methods , 2007, Pattern Recognit..

[32]  Tamás Szirányi,et al.  User-adaptive hand gesture recognition system with interactive training , 2005, Image Vis. Comput..

[33]  Saliha Aouat,et al.  Shapes Matching and Indexing Using Textual Descriptors , 2011 .

[34]  Hermann Hienz,et al.  Relevant features for video-based continuous sign language recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[35]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[36]  Mohammad A. Al-Rousan,et al.  Automatic Recognition of Arabic Sign Language Finger Spelling , 2001, Int. J. Comput. Their Appl..

[37]  Mohammed Waleed Kadous,et al.  Machine Recognition of Auslan Signs Using PowerGloves: Towards Large-Lexicon Recognition of Sign Lan , 1996 .

[38]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[39]  J. Flusser,et al.  Moments and Moment Invariants in Pattern Recognition , 2009 .

[40]  Sim Heng Ong,et al.  Image Analysis by Tchebichef Moments , 2001, IEEE Trans. Image Process..

[41]  Ramakrishnan Mukundan,et al.  Image quality assessment by discrete orthogonal moments , 2010, Pattern Recognit..

[42]  S Padam Priyal,et al.  A study on static hand gesture recognition using moments , 2010, 2010 International Conference on Signal Processing and Communications (SPCOM).

[43]  Saliha Aouat,et al.  Silhouettes Retrieval based on the Quad-tree Structure following an XML Description , 2012 .

[44]  Saliha Aouat,et al.  Accurate Similarity Measures for Silhouettes Recognition , 2012, VISAPP.

[45]  Chin-Chen Chang,et al.  New Approach for Static Gesture Recognition , 2006, J. Inf. Sci. Eng..

[46]  Mohamed F. Tolba,et al.  Neutralizing lighting non-homogeneity and background size in PCNN image signature for Arabic Sign Language recognition , 2011, The 2011 International Conference on Computer Engineering & Systems.

[47]  Lale Akarun,et al.  A belief-based sequential fusion approach for fusing manual signs and non-manual signals , 2009, Pattern Recognit..

[48]  Agnès Just,et al.  Hand Posture Classification and Recognition using the Modified Census Transform , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[49]  Saliha Aouat,et al.  Shape matching using coarse descriptors , 2010, Int. J. Comput. Vis. Robotics.

[50]  Saliha Aouat,et al.  Contours Matching with A Text-based Method , 2013 .

[51]  Kar-Seng Loke,et al.  Image reconstruction using various discrete orthogonal polynomials in comparison with DCT , 2007, Appl. Math. Comput..

[52]  Saliha Aouat,et al.  A Machine learning approach for Shape From Shading , 2016, ArXiv.

[53]  Saliha Aouat.  Shape Codification Indexing and Retrieval Using the Quad-Tree Structure , 2013, Int. J. Comput. Vis. Image Process..

[54]  Karl-Friedrich Kraiss,et al.  Video-based sign recognition using self-organizing subunits , 2002, Object recognition supported by user interaction for service robots.