Recognizing fingerspelling in SIBI (sistem isyarat bahasa Indonesia) using OpenPose and elliptical fourier descriptor

Humans are social beings; they communicate with other humans to coexist. In general, humans use verbal methods to communicate. However, the limited of verbal communication in people with hearing loss causes them to use non-verbal communication through sign language. It is challenging for nondisabled to grasp sign language quickly because it takes a long time to learn and understand. Therefore, a system that can recognize SIBI is needed. This research focused on fingerspelling gesture in SIBI, concentrating on finger and hand movements. The accuracy results by using OpenPose as feature extraction achieved a 61.49% accuracy. To improve accuracy, by using OpenPose as hand tracking, image masking as pre-processing, and Elliptical Fourier Descriptor as feature extraction, this method achieved a 64.05% accuracy. We investigated whether feature extraction with EFD and OpenPose for hand tracking can improve accuracy; however, some labels still could not be detected correctly. To improve accuracy, image masking and delete similarity frames are used as preprocessing; OpenPose uses as hand tracking, and EFD uses as feature extraction. This method achieved 67.23% accuracy.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Alfan Farizki Wicaksono,et al.  Recognition of Sign Language System for Indonesian Language Using Long Short-Term Memory Neural Networks , 2018 .

[3]  Erdefi Rakun,et al.  Spectral domain cross correlation function and generalized Learning Vector Quantization for recognizing and classifying Indonesian Sign Language , 2012, 2012 International Conference on Advanced Computer Science and Information Systems (ICACSIS).

[4]  Erdefi Rakun,et al.  Human Skeleton Feature Extraction from 2-Dimensional Video of Indonesian Language Sign System (SIBI [Sistem Isyarat Bahasa Indonesia]) Gestures , 2019, ICCAI '19.

[5]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Yaser Sheikh,et al.  Hand Keypoint Detection in Single Images Using Multiview Bootstrapping , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  C. R. Biologies,et al.  liptical Fourier descriptors for contours in three dimensions : new tool for morphometrical analysis in biology scripteurs elliptiques de Fourier des contours en trois dimensions : nouvel outil pour l ’ analyse morphométrique en biologie , 2012 .

[8]  Erdefi Rakun,et al.  Feature Extraction from Smartphone Images by Using Elliptical Fourier Descriptor, Centroid and Area for Recognizing Indonesian Sign Language SIBI (Sistem Isyarat Bahasa Indonesia) , 2019, 2019 2nd International Conference on Intelligent Autonomous Systems (ICoIAS).