Non-Invasive Silent Speech Recognition in Multiple Sclerosis with Dysphonia

We present the first non-invasive real-time silent speech system that helps patients with speech disorders to communicate in natural language voicelessly, merely by articulating words or sentences in the mouth without producing any sounds. We collected neuromuscular recordings to build a dataset of 10 trials of 15 sentences from each of 3 multiple sclerosis (MS) patients with dysphonia, spanning a range of severity and subsequently affected speech attributes. We present a pipeline wherein we carefully preprocess the data, develop a convolutional neural architecture and employ personalized machine learning. Our system achieves a mean overall test accuracy of 0.81 at a mean information transfer rate of 203.73 bits per minute averaged over all patients. Our work demonstrates the potential of a reliable and promising human-computer interface that classifies intended sentences from silent speech and hence, paves the path for future work with further speech disorders in conditions such as amyotrophic lateral sclerosis (ALS), stroke, and oral cancer, among others.

[1]  Angelina Leigh,et al.  The Experiences of Intimacy for Adults with Acquired Communication Disorders Using Augmentative and Alternative Communication (AAC) , 2010 .

[2]  M. S. Hawley,et al.  A Voice-Input Voice-Output Communication Aid for People With Severe Speech Impairment , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[3]  Arnav Kapur,et al.  Human-machine cognitive coalescence through an internal duplex interface , 2018 .

[4]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[5]  Placido Bramanti,et al.  Augmentative and alternative communication improves quality of life in the early stages of amyotrophic lateral sclerosis. , 2019, Functional neurology.

[6]  J H Verbeek,et al.  Interventions for preventing voice disorders in adults. , 2007, The Cochrane database of systematic reviews.

[7]  Douglas D. O'Shaughnessy,et al.  Alternative Speech Communication System for Persons with Severe Speech Disorders , 2009, EURASIP J. Adv. Signal Process..

[8]  Alphi Elackattu,et al.  Palliative treatment of dysphonia and dysarthria. , 2009, Otolaryngologic clinics of North America.

[9]  Maysam Ghovanloo,et al.  Toward Silent-Speech Control of Consumer Wearables , 2015, Computer.

[10]  David R. Beukelman,et al.  Augmentative and alternative communication for adults with acquired severe communication disorders , 1988 .

[11]  Yoshua Bengio,et al.  Learning deep physiological models of affect , 2013, IEEE Computational Intelligence Magazine.

[12]  Gábor Gosztolya,et al.  Multi-Task Learning of Speech Recognition and Speech Synthesis Parameters for Ultrasound-based Silent Speech Interfaces , 2018, INTERSPEECH.

[13]  Tanja Schultz,et al.  Array-based Electromyographic Silent Speech Interface , 2013, BIOSIGNALS.

[14]  Pattie Maes,et al.  AlterEgo: A Personalized Wearable Silent Speech Interface , 2018, IUI.

[15]  Seyed Reza Shahamiri,et al.  Adaptive neuro-fuzzy inference system for evaluating dysarthric automatic speech recognition (ASR) systems: a case study on MVML-based ASR , 2019, Soft Comput..

[16]  Shirley A. Brown,et al.  Swallowing and Speaking Challenges for the MS Patient , 2000 .

[17]  Phil D. Green,et al.  Automatic speech recognition with sparse training data for dysarthric speakers , 2003, INTERSPEECH.

[18]  G Pfurtscheller,et al.  EEG-based communication: improved accuracy by response verification. , 1998, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[19]  Myungjong Kim,et al.  Speaker-Independent Silent Speech Recognition From Flesh-Point Articulatory Movements Using an LSTM Neural Network , 2017, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[20]  Karen J. Reynolds,et al.  Removing power line noise from recorded EMG , 2001, 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[21]  J. M. Gilbert,et al.  Silent speech interfaces , 2010, Speech Commun..

[22]  Alexandros Pino,et al.  Augmentative and Alternative Communication Systems for the Motor Disabled , 2014 .