Sign Language Phoneme Transcription with Rule-based Hand Trajectory Segmentation

A common approach to extract phonemes of sign language is to use an unsupervised clustering algorithm to group the sign segments. However, simple clustering algorithms based on distance measures usually do not work well on temporal data and require complex algorithms. In this paper, we present a simple and effective approach to extract phonemes from American sign language sentences. We first apply a rule-based segmentation algorithm to segment the hand motion trajectories of signed sentences. We then extract feature descriptors based on principal component analysis to represent the segments efficiently. The segments are clustered by k-means using these high level features to derive phonemes. 25 different continuously signed sentences from a deaf signer were used to perform the analysis. After phoneme transcription, we trained Hidden Markov Models to recognize the sequence of phonemes in the sentences. Overall, our automatic approach yielded 165 segments, and 58 phonemes were obtained based on these segments. The average number of recognition errors was 18.8 (11.4%). In comparison, completely manual trajectory segmentation and phoneme transcription, involving considerable labor yielded 173 segments, 57 phonemes, and the average number of recognition errors was 33.8 (19.5%).