Supporting Rhythm Activities of Deaf Children using Music-Sensory-Substitution Systems

Rhythm is the first musical concept deaf people learn in music classes. However, hearing loss limits the amount of information that allows a deaf person to evaluate his or her performance and stay in sync with other musicians. In this paper, we investigated how a visual and vibrotactile music-sensory-substitution device, MuSS-Bits++, affects rhythm discrimination, reproduction, and expressivity of deaf people. We conducted a controlled study with 11 deaf children and found that most participants felt more confident wearing the device in vibration mode even when it did not objectively improve their accuracy. Furthermore, we studied how MuSS-Bits++ can be used in music classes at deaf schools and what challenges and opportunities arise in such a setting. Based on these studies, we discuss insights and future directions that support the design and development of music-sensory-substitution systems for music making.

[1]  Peter Essens,et al.  Perception of Temporal Patterns , 1985 .

[2]  B. Einspruch.  Deaf in America: Voices From a Culture , 1989 .

[3]  Lars C. Wolf,et al.  Enabling network-centric music performance in wide-area networks , 2006, CACM.

[4]  Carl E. Seashore.  Music and the Deaf , 1939 .

[5]  G. von Békésy,et al.  Similarities between hearing and skin sensations. , 1959, Psychological review.

[6]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[7]  Michael H. Thaut,et al.  Music Listening and Music Making in the Treatment of Neurological Disorders and Impairments , 2010 .

[8]  H. Bastian.  Sensation and Perception.—I , 1869, Nature.

[9]  Qian-Jie Fu,et al.  Melodic Contour Identification and Music Perception by Cochlear Implant Users , 2009, Annals of the New York Academy of Sciences.

[10]  Eric Thul,et al.  Measuring the Complexity of Musical Rhythm , 2008 .

[11]  Deborah I. Fels,et al.  Designing the Model Human Cochlea: An Ambient Crossmodal Audio-Tactile Display , 2009, IEEE Transactions on Haptics.

[12]  Josh D. Tenenberg,et al.  A blind person's interactions with technology , 2009, Commun. ACM.

[13]  Graham J Hole,et al.  How to Design and Report Experiments , 2002 .

[14]  Suranga Nanayakkara,et al.  MuSS-bits: sensor-display blocks for deaf people to explore musical sounds , 2016, OZCHI.

[15]  Ómar I. Jóhannesson,et al.  Designing sensory-substitution devices: Principles, pitfalls and potential , 2016, Restorative neurology and neuroscience.

[16]  Godfried T. Toussaint,et al.  A Comparison of Rhythmic Similarity Measures , 2004, ISMIR.

[17]  S. Huffel,et al.  Influence of Mental Stress on Heart Rate and Heart Rate Variability , 2009 .

[18]  Brian C. J. Moore,et al.  Music and Hearing Aids , 2014, Trends in hearing.

[19]  Suranga Nanayakkara,et al.  Ad-Hoc Access to Musical Sound for Deaf Individuals , 2016, ASSETS.

[20]  D. Levitin.  This is Your Brain on Music: Understanding a Human Obsession , 2019 .

[21]  Andrew Sears,et al.  Representing users in accessibility research , 2012, TACC.

[22]  Suranga Nanayakkara,et al.  An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair , 2009, CHI.

[23]  Fabrizio Leo,et al.  Independent mechanisms for ventriloquism and multisensory integration as revealed by theta‐burst stimulation , 2010, The European journal of neuroscience.

[24]  Paul Bach-Y-Rita,et al.  Closing an open-loop control system: vestibular substitution through the tongue. , 2003, Journal of integrative neuroscience.

[25]  Ye Wang,et al.  MOGAT: mobile games with auditory training for children with cochlear implants , 2012, ACM Multimedia.

[26]  Alice-Ann Darrow,et al.  The Role of Music in Deaf Culture: Implications for Music Educators , 1993 .

[27]  Jay T Rubinstein,et al.  Music perception in cochlear implant users and its relationship with psychophysical capabilities. , 2008, Journal of rehabilitation research and development.

[28]  Gottfried Schlaug,et al.  Musicians and music making as a model for the study of brain plasticity. , 2015, Progress in brain research.

[29]  Suranga Nanayakkara,et al.  Scaffolding the Music Listening and Music Making Experience for the Deaf , 2018 .

[30]  Deborah I. Fels,et al.  The emoti-chair: an interactive tactile music exhibit , 2010, CHI Extended Abstracts.