Dialogue enabling speech-to-text user assistive agent with auditory perceptual beamforming for hearing-impaired

A novel approach for assisting effective bidirectional communication of intentions between people of normal hearing and hearing-impaired is presented in this paper. In particular, we demonstrate the intelligent hearing assistive utility by incorporating a speech-to-text interface such that the intended speaker's speech is directionally contained by means of a multichannel acoustic beamformer. Clear understanding of the normal hearing person's speech can reduce the burden on the hearing impaired in terms of required explicit response under context sensitive dialogue scenarios. Hence, hearing impaired friendlier interface device perceiving the utterances of normal hearing person is highly desirable. The proposed interface is a portable device with directional sound selection capability that performs speech-to-text task of normal hearing person's speech in diverse dialogue contexts. The relevant experimental results have confirmed that the proposed interface design is a feasible approach for realizing an effective and efficient intelligent agent, as it reduces the amount of work required from hearing impaired user who is interacting with the normal hearing people.