MuSS-bits: sensor-display blocks for deaf people to explore musical sounds

Hearing loss makes learning a musical instrument a challenging task. Prior work suggests that a universal sensory substitution system that works uniformly across all deaf users may not exist given the diversity within the deaf community. In this paper, we present Music Sensory Substitution (MuSS) Bits, wireless sensor-display pairs that enable exploration of musical sound as well as customization of visual and vibrotactile feedback to cater to individual requirements and preferences. MuSS-Bits are portable, easy to deploy on the user's body, on an instrument, or in the environment, and provide real-time feedback. We review existing music sensory substitution systems, discuss the design space for MuSS-Bits, present details of a prototypical implementation and illustrate interaction possibilities including initial user reactions.

[1]  Densil Cabrera,et al.  Seeing sound: real-time sound visualisation in visual feedback loops used for training musicians , 2005, Ninth International Conference on Information Visualisation (IV'05).

[2]  Chitta Saha,et al.  Modeling and experimental investigation of an AA-sized electromagnetic generator for harvesting energy from human motion , 2008, Smart Materials and Structures.

[3]  Fabrizio Leo,et al.  Independent mechanisms for ventriloquism and multisensory integration as revealed by theta‐burst stimulation , 2010, The European journal of neuroscience.

[4]  David W. Fourney,et al.  Can Computer Representations of Music Enhance Enjoyment for Individuals Who Are Hard of Hearing? , 2012, ICCHP.

[5]  Marcelo M. Wanderley,et al.  Vibrotactile Feedback in Digital Musical Instruments , 2006, NIME.

[6]  J G Clark,et al.  Uses and abuses of hearing loss classification. , 1981, ASHA.

[7]  Jay T Rubinstein,et al.  Music perception in cochlear implant users and its relationship with psychophysical capabilities. , 2008, Journal of rehabilitation research and development.

[8]  Deborah I. Fels,et al.  Towards a model human cochlea: sensory substitution for crossmodal audio-tactile displays , 2008, Graphics Interface.

[9]  Suranga Nanayakkara,et al.  An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair , 2009, CHI.

[10]  Victoria S. Hagedorn,et al.  Musical Learning for Hearing Impaired Children. , 1992 .

[11]  Chris Harrison,et al.  Unlocking the expressivity of point lights , 2012, CHI.

[12]  Deborah I. Fels,et al.  Designing the Model Human Cochlea: An Ambient Crossmodal Audio-Tactile Display , 2009, IEEE Transactions on Haptics.

[13]  Ryan M. Traylor,et al.  A Haptic Back Display for Attentional and Directional Cueing , 2003 .

[14]  Ellen Yi-Luen Do,et al.  Mobile music touch: mobile tactile stimulation for passive learning , 2010, CHI.

[15]  Deborah I. Fels,et al.  Modelling Perceptual Elements of Music in a Vibrotactile Display for Deaf Users: A Field Study , 2009, 2009 Second International Conferences on Advances in Computer-Human Interactions.

[16]  Deborah I. Fels,et al.  Expressing Emotions Using Animated Text Captions , 2006, ICCHP.

[17]  Eric J. Isaacson.  What You See Is What You Get: on Visualizing Music , 2005, ISMIR.

[18]  Suranga Nanayakkara,et al.  Enhancing Musical Experience for the Hearing-Impaired Using Visual and Haptic Displays , 2012, Hum. Comput. Interact..

[19]  Deborah I. Fels,et al.  Seeing the music can animated lyrics provide access to the emotional content in music for people who are deaf or hard of hearing? , 2009, 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH).

[20]  Elizabeth May.  Music for Deaf Children , 1961 .

[21]  Brian C. J. Moore,et al.  Music and Hearing Aids , 2014, Trends in hearing.

[22]  Ye Wang,et al.  MOGAT: mobile games with auditory training for children with cochlear implants , 2012, ACM Multimedia.

[23]  Marcelo M. Wanderley,et al.  A Systematic Approach to Musical vibrotactile feedback , 2007, ICMC.

[24]  Alice-Ann Darrow,et al.  The Role of Music in Deaf Culture: Implications for Music Educators , 1993 .

[25]  Tara Matthews,et al.  Visualizing non-speech sounds for the deaf , 2005, Assets '05.

[26]  Josh D. Tenenberg,et al.  A blind person's interactions with technology , 2009, Commun. ACM.

[27]  Yun-Long Lay,et al.  Development and evaluation of computer-aided music-learning system for the hearing impaired , 2007, J. Comput. Assist. Learn..

[28]  Phillip M. Hash.  Teaching Instrumental Music to Deaf and Hard of Hearing Students , 2003 .

[29]  Deborah I. Fels,et al.  EnACT: A Software Tool for Creating Animated Text Captions , 2008, ICCHP.

[30]  Deborah I. Fels,et al.  The emoti-chair: an interactive tactile music exhibit , 2010, CHI Extended Abstracts.

[31]  James A. Landay,et al.  Can you see what i hear?: the design and evaluation of a peripheral sound display for the deaf , 2003, CHI '03.

[32]  Norman I. Badler,et al.  Movies from music: Visualizing musical compositions , 1979, SIGGRAPH '79.

[33]  C M Reed,et al.  Analytic study of the Tadoma method: further experiments with inexperienced observers. , 1982, Journal of speech and hearing research.

[34]  Tom Yeh,et al.  Seen music: ambient music data visualization for children with hearing impairments , 2015, IDC.

[35]  Bruno La Versa,et al.  MUVIB: music and vibration , 2014, ISWC '14 Adjunct.

[36]  Deborah I. Fels,et al.  Creating access to music through visualization , 2009, 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH).

[37]  S Levy-Tzedek,et al.  Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. , 2012, Restorative neurology and neuroscience.

[38]  J. Medina.  Brain Rules: 12 Principles for Surviving and Thriving at Work, Home, and School , 2008 .

[39]  Deborah I. Fels,et al.  Creating an Entertaining and Informative Music Visualization , 2012, ICCHP.

[40]  R. T. Verrillo,et al.  Vibration Sensation in Humans , 1992 .

[41]  Qian-Jie Fu,et al.  Melodic Contour Identification and Music Perception by Cochlear Implant Users , 2009, Annals of the New York Academy of Sciences.

[42]  Jon Froehlich,et al.  Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing , 2015, CHI.

[43]  Lars C. Wolf,et al.  Enabling network-centric music performance in wide-area networks , 2006, CACM.

[44]  Deborah I. Fels,et al.  Playing vibrotactile music: A comparison between the Vibrochord and a piano keyboard , 2014, Int. J. Hum. Comput. Stud..