Haptic Voice Recognition: Augmenting speech modality with touch events for efficient speech recognition

This paper proposes the Haptic Voice Recognition (HVR), a multi-modal interface that combines speech and touch sensory inputs to perform voice recognition. These touch inputs form a series of haptic events that provide cues or ‘landmarks’ for word boundaries. These word boundary cues greatly reduce the search space for speech recognition, thereby making the decoding process more efficient and suitable for portable devices with limited compute and memory resources. Furthermore, having the knowledge of word boundaries also suppresses insertion and deletion errors. This is particularly helpful when recognition is performed in noisy environment. In this paper, a series of experiments were conducted to study the feasibility of augmenting touch events to automatic speech recognition and explore its potential benefits. Experiments were conducted with syntactically simulated haptic events on the Wall Street Journal database as well as realistic haptic events acquired using a prototype HVR interface implemented on a touchscreen device.