Brain-computer interfaces for speech communication

This paper briefly reviews current silent speech methodologies for normal and disabled individuals. Current techniques utilizing electromyographic (EMG) recordings of vocal tract movements are useful for physically healthy individuals but fail for tetraplegic individuals who do not have accurate voluntary control over the speech articulators. Alternative methods utilizing EMG from other body parts (e.g., hand, arm, or facial muscles) or electroencephalography (EEG) can provide capable silent communication to severely paralyzed users, though current interfaces are extremely slow relative to normal conversation rates and require constant attention to a computer screen that provides visual feedback and/or cueing. We present a novel approach to the problem of silent speech via an intracortical microelectrode brain computer interface (BCI) to predict intended speech information directly from the activity of neurons involved in speech production. The predicted speech is synthesized and acoustically fed back to the user with a delay under 50 ms. We demonstrate that the Neurotrophic Electrode used in the BCI is capable of providing useful neural recordings for over 4 years, a necessary property for BCIs that need to remain viable over the lifespan of the user. Other design considerations include neural decoding techniques based on previous research involving BCIs for computer cursor or robotic arm control via prediction of intended movement kinematics from motor cortical signals in monkeys and humans. Initial results from a study of continuous speech production with instantaneous acoustic feedback show the BCI user was able to improve his control over an artificial speech synthesizer both within and across recording sessions. The success of this initial trial validates the potential of the intracortical microelectrode-based approach for providing a speech prosthesis that can allow much more rapid communication rates.

[1]  P.R. Kennedy,et al.  Computer control using human intracortical local field potentials , 2004, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[2]  Michael J. Black,et al.  Assistive technology and robotic control using motor cortex ensemble‐based neural interface systems in humans with tetraplegia , 2007, The Journal of physiology.

[3]  K. E. Jones,et al.  A glass/silicon composite intracortical electrode array , 2006, Annals of Biomedical Engineering.

[4]  Tanja Schultz,et al.  Automatic Speech Recognition Based on Electromyographic Biosignals , 2008, BIOSTEC.

[5]  E. W. Sellers,et al.  Toward enhanced P300 speller performance , 2008, Journal of Neuroscience Methods.

[6]  H. Flor,et al.  A spelling device for the paralysed , 1999, Nature.

[7]  R. Ilmoniemi,et al.  Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain , 1993 .

[8]  J. A. Wilson,et al.  Two-dimensional movement control using electrocorticographic signals in humans , 2008, Journal of neural engineering.

[9]  L. Maier-Hein,et al.  Session independent non-audible speech recognition using surface electromyography , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[10]  Jerald D. Kralik,et al.  Chronic, multisite, multielectrode recordings in macaque monkeys , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Tanja Schultz,et al.  EEG-based Speech Recognition - Impact of Temporal Effects , 2009, BIOSIGNALS.

[12]  Michael J. Black,et al.  Multi-state decoding of point-and-click control signals from motor cortical activity in a human with tetraplegia , 2007, 2007 3rd International IEEE/EMBS Conference on Neural Engineering.

[13]  D. G. MacKay Metamorphosis of a critical interval: age-linked changes in the delay in auditory feedback that produces maximal disruption of speech. , 1968, The Journal of the Acoustical Society of America.

[14]  E. M. Schmidt,et al.  Long-term chronic recording from cortical neurons , 1976, Experimental Neurology.

[15]  H. Flor,et al.  The thought translation device (TTD) for completely paralyzed patients. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[16]  Arthur Gelb,et al.  Applied Optimal Estimation , 1974 .

[17]  D.J. McFarland,et al.  The wadsworth BCI research and development program: at home with BCI , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[18]  R. Normann,et al.  Chronic recording capability of the Utah Intracortical Electrode Array in cat sensory cortex , 1998, Journal of Neuroscience Methods.

[19]  Emery N. Brown,et al.  Computational Neuroscience: A Comprehensive Approach , 2022 .

[20]  Makoto Sato,et al.  Single-trial classification of vowel speech imagery using common spatial patterns , 2009, Neural Networks.

[21]  Tanja Schultz,et al.  Sub-word unit based non-audible speech recognition using surface electromyography , 2006, INTERSPEECH.

[22]  Frank H. Guenther,et al.  A neural network model of speech acquisition and motor equivalent speech production , 2004, Biological Cybernetics.

[23]  J. Donoghue,et al.  Primary Motor Cortex Tuning to Intended Movement Kinematics in Humans with Tetraplegia , 2008, The Journal of Neuroscience.

[24]  David M. Santucci,et al.  Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates , 2003, PLoS biology.

[25]  R. J. Vetter,et al.  Silicon-substrate intracortical microelectrode arrays for long-term recording of neuronal spike activity in cerebral cortex , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[26]  P. Kennedy,et al.  Neurotrophic electrode: Method of assembly and implantation into human motor speech cortex , 2008, Journal of Neuroscience Methods.

[27]  U. Mitzdorf Current source-density method and application in cat cerebral cortex: investigation of evoked potentials and EEG phenomena. , 1985, Physiological reviews.

[28]  E Donchin,et al.  The mental prosthesis: assessing the speed of a P300-based brain-computer interface. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[29]  Kim Binsted,et al.  Small-vocabulary speech recognition using surface electromyography , 2006, Interact. Comput..

[30]  Philip Kennedy Comparing Electrodes for Use as Cortical Control Signals: Tiny Tines, Tiny Wires, or Tiny Cones on Wires Which Is Best? , 2007 .

[31]  F. Guenther,et al.  A Wireless Brain-Machine Interface for Real-Time Speech Synthesis , 2009, PloS one.

[32]  F H Guenther,et al.  Speech sound acquisition, coarticulation, and rate effects in a neural network model of speech production. , 1995, Psychological review.

[33]  N. Birbaumer,et al.  The thought-translation device (TTD): neurobehavioral mechanisms and clinical outcome , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[34]  P. Kennedy,et al.  The cone electrode: Ultrastructural studies following long-term recording in rat and monkey cortex , 1992, Neuroscience Letters.

[35]  Uri T Eden,et al.  A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. , 2005, Journal of neurophysiology.

[36]  K. Wise,et al.  A three-dimensional microelectrode array for chronic neural recording , 1994, IEEE Transactions on Biomedical Engineering.

[37]  Jerald D. Kralik,et al.  Real-time prediction of hand trajectory by ensembles of cortical neurons in primates , 2000, Nature.

[38]  Andrew S. Whitford,et al.  Cortical control of a prosthetic arm for self-feeding , 2008, Nature.

[39]  K. Wise,et al.  An integrated-circuit approach to extracellular microelectrodes. , 1970, IEEE transactions on bio-medical engineering.

[40]  D. D. Lee,et al.  Sub auditory speech recognition based on EMG signals , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[41]  S. Cobb Speech and Brain-Mechanisms. , 1960 .

[42]  A.K. Barros,et al.  Subvocal Speech Recognition Based on EMG Signal Using Independent Component Analysis and Neural Network MLP , 2008, 2008 Congress on Image and Signal Processing.

[43]  Linnea C. Ehri,et al.  The recognition of words , 1978 .

[44]  Xiaorong Gao,et al.  Design and implementation of a brain-computer interface with high transfer rates , 2002, IEEE Transactions on Biomedical Engineering.

[45]  Tanja Schultz,et al.  Towards Speaker-adaptive Speech Recognition based on Surface Electromyography , 2009, BIOSIGNALS.

[46]  J. Wolpaw,et al.  Towards an independent brain–computer interface using steady state visual evoked potentials , 2008, Clinical Neurophysiology.

[47]  A. B.,et al.  SPEECH COMMUNICATION , 2001 .

[48]  Gerwin Schalk,et al.  A brain–computer interface using electrocorticographic signals in humans , 2004, Journal of neural engineering.

[49]  P R Kennedy,et al.  Direct control of a computer from the human central nervous system. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[50]  J.P. Donoghue,et al.  Reliability of signals from a chronically implanted, silicon-based electrode array in non-human primate primary motor cortex , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[51]  Dean J Krusienski,et al.  A comparison of classification techniques for the P300 Speller , 2006, Journal of neural engineering.

[52]  J. M. Gilbert,et al.  Development of a (silent) speech recognition system for patients following laryngectomy. , 2008, Medical engineering & physics.

[53]  J. Wolpaw,et al.  A P300 event-related potential brain–computer interface (BCI): The effects of matrix size and inter stimulus interval on performance , 2006, Biological Psychology.

[54]  V. Mountcastle,et al.  Posterior parietal association cortex of the monkey: command functions for operations within extrapersonal space. , 1975, Journal of neurophysiology.

[55]  A P Georgopoulos,et al.  On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[56]  Satrajit S. Ghosh,et al.  Neural modeling and imaging of the cortical interactions underlying syllable production , 2006, Brain and Language.

[57]  D J McFarland,et al.  Brain-computer interface research at the Wadsworth Center. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[58]  P. Kennedy,et al.  Restoration of neural output from a paralyzed patient by a direct brain connection , 1998, Neuroreport.

[59]  Jonathan R Wolpaw,et al.  Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[60]  R A Normann,et al.  The Utah intracortical Electrode Array: a recording structure for potential brain-computer interfaces. , 1997, Electroencephalography and clinical neurophysiology.

[61]  Daryl R. Kipke,et al.  Wireless implantable microsystems: high-density electronic interfaces to the nervous system , 2004, Proceedings of the IEEE.

[62]  Cz. Basztura,et al.  Automatic speech recognition based on diphones , 1998, MELECON '98. 9th Mediterranean Electrotechnical Conference. Proceedings (Cat. No.98CH36056).

[63]  Tanja Schultz,et al.  Towards continuous speech recognition using surface electromyography , 2006, INTERSPEECH.

[64]  R A Bakay,et al.  Behavioral correlates of action potentials recorded chronically inside the Cone Electrode. , 1992, Neuroreport.

[65]  D. Kipke,et al.  Long-term neural recording characteristics of wire microelectrode arrays implanted in cerebral cortex. , 1999, Brain research. Brain research protocols.

[66]  Jianfeng Feng,et al.  Computational neuroscience , 1986, Behavioral and Brain Sciences.

[67]  Dawn M. Taylor,et al.  Direct Cortical Control of 3D Neuroprosthetic Devices , 2002, Science.

[68]  Jon A. Mukand,et al.  Neuronal ensemble control of prosthetic devices by a human with tetraplegia , 2006, Nature.

[69]  P Suppes,et al.  Brain wave recognition of words. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[70]  N. Birbaumer,et al.  The thought translation device: a neurophysiological approach to communication in total motor paralysis , 1999, Experimental Brain Research.

[71]  Frank H. Guenther,et al.  Artificial speech synthesizer control by brain-computer interface , 2009, INTERSPEECH.

[72]  P. Kennedy The cone electrode: a long-term electrode that records from neurites grown onto its recording surface , 1989, Journal of Neuroscience Methods.

[73]  Charles Jorgensen,et al.  Small Vocabulary Recognition Using Surface Electromyography in an Acoustically Harsh Environment , 2005 .

[74]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[75]  N. Birbaumer,et al.  A brain–computer interface (BCI) for the locked-in: comparison of different EEG classifications for the thought translation device , 2003, Clinical Neurophysiology.

[76]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[77]  L.J. Trejo,et al.  Brain-computer interfaces for 1-D and 2-D cursor control: designs using volitional control of the EEG spectrum or steady-state visual evoked potentials , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[78]  J. Wolpaw,et al.  Decoding two-dimensional movement trajectories using electrocorticographic signals in humans , 2007, Journal of neural engineering.