Appearance-Based Hand Sign Recognition from Intensity Image Sequences

Abstract In this paper, we present a new approach to recognizing hand signs. In this approach, motion recognition (the hand movement) is tightly coupled with spatial recognition (hand shape). The system uses multiclass, multidimensional discriminant analysis to automatically select the most discriminating linear features for gesture classification. A recursive partition tree approximator is proposed to do classification. This approach combined with our previous work on hand segmentation forms a new framework which addresses the three key aspects of hand sign interpretation: hand shape, location, and movement. The framework has been tested to recognize 28 different hand signs. The experimental results show that the system achieved a 93.2% recognition rate for test sequences that had not been used in the training phase. It is shown that our approach provide performance better than that of nearest neighbor classification in the eigensubspace.

[1]  R. F.,et al.  Mathematical Statistics , 1944, Nature.

[2]  W. Stokoe,et al.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[3]  Donald E. Knuth,et al.  Sorting and Searching , 1973 .

[4]  Y. Chien,et al.  Pattern classification and scene analysis , 1974 .

[5]  Richard J. Lipton,et al.  Multidimensional Searching Problems , 1976, SIAM J. Comput..

[6]  K. Nelson,et al.  Languages and Language-Related Skills in Deaf and Hearing Children , 2013 .

[7]  Harry Bornstein,et al.  The Signed English Starter , 1984 .

[8]  Antonin Guttman,et al.  R-trees: a dynamic index structure for spatial searching , 1984, SIGMOD '84.

[9]  Bernard Chazelle,et al.  How to Search in History , 1983, Inf. Control..

[10]  Christos Faloutsos,et al.  The R+-Tree: A Dynamic Index for Multi-Dimensional Objects , 1987, VLDB.

[11]  Nick Roussopoulos,et al.  Faloutsos: "the r+- tree: a dynamic index for multidimensional objects , 1987 .

[12]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[13]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Hans-Peter Kriegel,et al.  The R*-tree: an efficient and robust access method for points and rectangles , 1990, SIGMOD '90.

[15]  Fumio Kishino,et al.  Stereo-based description by generalized cylinder complexes from occluding contours , 1991, Systems and Computers in Japan.

[16]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[17]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Roberto Cipolla,et al.  Robust structure from motion using motion parallax , 1993, 1993 (4th) International Conference on Computer Vision.

[19]  Alex Pentland,et al.  Space-time gestures , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Stanley M. Dunn,et al.  Learning Shape Classes , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  John,et al.  On Comprehensive Visual Learning , 1994 .

[22]  Hiroshi Murase,et al.  Illumination planning for object recognition in structured environments , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Michael Isard,et al.  3D position, attitude and shape input using video tracking of hands and lips , 1994, SIGGRAPH.

[24]  Mubarak Shah,et al.  Visual gesture recognition , 1994 .

[25]  Charles Kervrann,et al.  A hierarchical statistical framework for the segmentation of deformable objects in image sequences , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Thomas S. Huang,et al.  Vision based hand modeling and tracking for virtual teleconferencing and telecollaboration , 1995, Proceedings of IEEE International Conference on Computer Vision.

[27]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[28]  Aaron F. Bobick,et al.  A state-based technique for the summarization and recognition of gesture , 1995, Proceedings of IEEE International Conference on Computer Vision.

[29]  Vladimir Pavlovic,et al.  Hand Gesture Modeling, Analysis, and Synthesis , 1995 .

[30]  Yuntao Cui,et al.  Learning-based hand sign recognition using SHOSLIF-M , 1995, Proceedings of IEEE International Conference on Computer Vision.

[31]  Juyang Weng,et al.  Using Discriminant Eigenfeatures for Image Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Yuntao Cui,et al.  Hand segmentation using learning-based prediction and verification for hand sign recognition , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Narendra Ahuja,et al.  Extracting gestural motion trajectories , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[34]  Michael J. Black,et al.  Recognizing temporal trajectories using the condensation algorithm , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[35]  Yuntao Cui,et al.  A Learning-Based Prediction-and-Verification Segmentation Scheme for Hand Sign Image Sequence , 1999, IEEE Trans. Pattern Anal. Mach. Intell..