Selfie video based continuous Indian sign language recognition system

Abstract This paper introduces a novel method to bring sign language closer to real time application on mobile platforms. Selfie captured sign language video is processed by constraining its computing power to that of a smart phone. Pre-filtering, segmentation and feature extraction on video frames creates a sign language feature space. Minimum Distance and Artificial Neural Network classifiers on the sign feature space is trained and tested iteratively. Sobel edge operator's power is enhanced with morphology and adaptive thresholding giving a near perfect segmentation of hand and head portions compensating for the small vibrations of the selfie stick. Word matching score (WMS) gives the performance of the proposed method with an average WMS of around 85.58% for MDC and 90% for ANN with a small variation of 0.3 s in classification times. Neural network classifiers with fast training algorithms will certainly make this novel selfie sign language recognizer application into app stores.