Haptic Conversion Using Detected Sound Event in Home Monitoring System for the Hard-of-Hearing

In this paper, we present a haptic conversion method using detected sound event in home monitoring assistant system for the deaf and hard-of-hearing. In the home monitoring assistant system, sounds generated in the home environment are automatically detected by sound event detection based on wireless acoustic sensor networks. And the detected sound event is converted into text and haptic vibration, and provided to help people with hearing loss. The proposed approach is mainly composed of four modules, including signal estimation based on packet loss concealment, reliable sensor channel selection using a multi-channel cross-correlation coefficient, sound event detection using bidirectional gated recurrent neural networks, and sound-to-haptic effect conversion based on percussive-harmonic source separation. Experiments show that the hard-of-hearing users were receptive to the effect of the proposed haptic conversion using the sound event detected by bidirectional gated recurrent neural networks in the home monitoring system.

[1]  Changchun Bao,et al.  Speech enhancement using generalized weighted β-order spectral amplitude estimator , 2014, Speech Commun..

[2]  Athanasios Mouchtaris,et al.  Real-Time Multiple Sound Source Localization and Counting Using a Circular Microphone Array , 2013, IEEE Transactions on Audio, Speech, and Language Processing.

[3]  Antoine Liutkus,et al.  Kernel Additive Models for Source Separation , 2014, IEEE Transactions on Signal Processing.

[4]  Derry Fitzgerald,et al.  Harmonic/Percussive Separation Using Median Filtering , 2010 .

[5]  Heikki Huttunen,et al.  Multi-label vs. combined single-label sound event detection with deep neural networks , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[6]  Deborah I. Fels,et al.  Designing the Model Human Cochlea: An Ambient Crossmodal Audio-Tactile Display , 2009, IEEE Transactions on Haptics.

[7]  Prachi Sharma.  Wireless Sensor Networks for Environmental Monitoring , 2014 .

[8]  Heikki Huttunen,et al.  Polyphonic sound event detection using multi label deep neural networks , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[9]  Annamaria Mesaros,et al.  Metrics for Polyphonic Sound Event Detection , 2016 .

[10]  Tuomas Virtanen,et al.  TUT database for acoustic scene classification and sound event detection , 2016, 2016 24th European Signal Processing Conference (EUSIPCO).

[11]  Yao Liang.  Wireless Sensor Networks for Environmental Monitoring , 2015 .

[12]  Abderrahmane Amrouche,et al.  An improved packet loss concealment technique for speech transmission in VOIP , 2018, 2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP).

[13]  Andreas M. Ali,et al.  Acoustic monitoring in terrestrial environments using microphone arrays: applications, technological considerations and prospectus , 2011 .

[14]  Seungmoon Choi,et al.  Real-time perception-level translation from audio signals to vibrotactile effects , 2013, CHI.

[15]  Eckehard G. Steinbach,et al.  Low bitrate source-filter model based compression of vibrotactile texture signals in haptic teleoperation , 2012, ACM Multimedia.

[16]  Franz Pernkopf,et al.  Gated Recurrent Networks applied to Acoustic Scene Classification , 2016, DCASE.

[17]  Jill Fain Lehman,et al.  Channel selection based on multichannel cross-correlation coefficients for distant speech recognition , 2011, 2011 Joint Workshop on Hands-free Speech Communication and Microphone Arrays.

[18]  Matthias Wölfel.  Channel selection by class separability measures for automatic transcriptions on distant microphones , 2007, INTERSPEECH.

[19]  Yoshua Bengio,et al.  Gated Feedback Recurrent Neural Networks , 2015, ICML.