Boris: a Spoken Conversational Agent for Music Production for People with Motor Disabilities

Previous studies suggest that engagement in musical activities may enhance well-being and impact social inclusion. However, unfortunately, people with physical disabilities cannot often use musical instruments or music production software due to accessibility issues. We propose Boris, an original conversational agent specific for people with a physical disability, to entertain, stimulate expressiveness, and promote communication. Boris enables (even inexperienced) users to compose songs through hands-free interaction by analyzing their vocalizations to obtain more than just their transcription: the system listens to the user even while humming a song and generates a melody by learning and reproducing their human voice patterns. Indeed, it exploits an artificial musical intelligence that can imitate the typically human cognitive skills to produce music using an advanced technique called abstract melody.

[1]  Patricia D. Mautone,et al.  Social cues in multimedia learning: Role of speaker's voice. , 2003 .

[2]  T. Ingram,et al.  A Description and Classification of Common Speech Disorders Associated With Cerebral Palsy , 1961, Cerebral palsy bulletin.

[3]  S. Alper,et al.  Assistive Technology for Individuals with Disabilities: A Review and Synthesis of the Literature , 2006 .

[4]  E. Bertini,et al.  Speech and Language Disorders in Friedreich Ataxia: Highlights on Phenomenology, Assessment, and Therapy , 2019, The Cerebellum.

[5]  Rebecca Fiebrink,et al.  Sound Control: Supporting Custom Musical Interface Design for Children with Disabilities , 2019, NIME.

[6]  Anthony J. Hornof,et al.  EyeMusic: Making Music with the Eyes , 2004, NIME.

[7]  Sara K. Jones.  Teaching Students with Disabilities , 2015 .

[8]  L. Ramig,et al.  Speech and swallowing disorders in Parkinson disease , 2008, Current opinion in otolaryngology & head and neck surgery.

[9]  Ricardo Pagán-Rodríguez,et al.  How do disabled individuals spend their leisure time? , 2014, Disability and health journal.

[10]  Tim Anderson,et al.  “Composability”: widening participation in music making for people with disabilities via music software and controller solutions , 1996, Assets '96.

[11]  Jong-Tae Lim,et al.  Robotic smart house to assist people with movement disabilities , 2007, Auton. Robots.

[12]  Marc A. C. Sarazin,et al.  Singing and social inclusion , 2014, Front. Psychol..

[13]  Fabio Catania,et al.  Emoty: an Emotionally Sensitive Conversational Agent for People with Neurodevelopmental Disorders , 2019, HICSS.

[14]  Emma Frid,et al.  Accessible Digital Musical Instruments - A Review of Musical Interfaces in Inclusive Music Practice , 2019, Multimodal Technol. Interact..

[15]  Fabio Catania,et al.  Musical and Conversational Artificial Intelligence , 2020, IUI Companion.

[16]  Micol Spitale,et al.  CORK: A COnversational agent framewoRK exploiting both rational and emotional intelligence , 2019, IUI Workshops.

[17]  Jacob Harrison,et al.  An adapted bass guitar for one-handed playing , 2017, NIME.

[18]  J. Walecki,et al.  A comparison of spastic diplegic and tetraplegic cerebral palsy. , 2005, Pediatric neurology.

[19]  J. McComas,et al.  Virtual Reality Applications for Prevention, Disability Awareness, and Physical Therapy Rehabilitation in Neurology: Our Recent Work , 2002 .

[20]  Thomas B. Moeslund,et al.  The Prospects of Musical Instruments For People with Physical Disabilities , 2016, NIME.

[21]  T. Hasselbring,et al.  Use of computer technology to help students with special needs. , 2000, The Future of children.

[22]  Leah Findlater,et al.  "Accessibility Came by Accident": Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities , 2018, CHI.

[23]  Miguel Ortiz,et al.  Bespoke Design for Inclusive Music: The Challenges of Evaluation , 2019, NIME.

[24]  Karalyn Patterson,et al.  Phonological and Articulatory Impairment in Alzheimer's Disease: A Case Series , 2000, Brain and Language.