Classification of Intended Phoneme Production from Chronic Intracortical Microelectrode Recordings in Speech-Motor Cortex

We conducted a neurophysiological study of attempted speech production in a paralyzed human volunteer using chronic microelectrode recordings. The volunteer suffers from locked-in syndrome leaving him in a state of near-total paralysis, though he maintains good cognition and sensation. In this study, we investigated the feasibility of supervised classification techniques for prediction of intended phoneme production in the absence of any overt movements including speech. Such classification or decoding ability has the potential to greatly improve the quality-of-life of many people who are otherwise unable to speak by providing a direct communicative link to the general community. We examined the performance of three classifiers on a multi-class discrimination problem in which the items were 38 American English phonemes including monophthong and diphthong vowels and consonants. The three classifiers differed in performance, but averaged between 16 and 21% overall accuracy (chance-level is 1/38 or 2.6%). Further, the distribution of phonemes classified statistically above chance was non-uniform though 20 of 38 phonemes were classified with statistical significance for all three classifiers. These preliminary results suggest supervised classification techniques are capable of performing large scale multi-class discrimination for attempted speech production and may provide the basis for future communication prostheses.

[1]  R. Fisher THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[2]  S. Cobb Speech and Brain-Mechanisms. , 1960 .

[3]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .

[4]  K. Wise,et al.  An integrated-circuit approach to extracellular microelectrodes. , 1970, IEEE transactions on bio-medical engineering.

[5]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[6]  A W Hahn,et al.  The adhesion of glow--discharge polymers, Silastic and Parylene to implantable platinum electrodes: results of tensile pull tests after exposure to isotonic sodium chloride. , 1981, Biomaterials.

[7]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[8]  P. Kennedy The cone electrode: a long-term electrode that records from neurites grown onto its recording surface , 1989, Journal of Neuroscience Methods.

[9]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[10]  Erich E. Sutter,et al.  The brain response interface: communication through visually-induced electrical brain responses , 1992 .

[11]  R A Bakay,et al.  Behavioral correlates of action potentials recorded chronically inside the Cone Electrode. , 1992, Neuroreport.

[12]  P. Kennedy,et al.  The cone electrode: Ultrastructural studies following long-term recording in rat and monkey cortex , 1992, Neuroscience Letters.

[13]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[14]  K. Wise,et al.  A three-dimensional microelectrode array for chronic neural recording , 1994, IEEE Transactions on Biomedical Engineering.

[15]  R. Tibshirani,et al.  Flexible Discriminant Analysis by Optimal Scoring , 1994 .

[16]  M. Jeannerod,et al.  Possible involvement of primary motor cortex in mentally simulated movement: a functional magnetic resonance imaging study. , 1996, Neuroreport.

[17]  R A Normann,et al.  The Utah intracortical Electrode Array: a recording structure for potential brain-computer interfaces. , 1997, Electroencephalography and clinical neurophysiology.

[18]  M S Lewicki,et al.  A review of methods for spike sorting: the detection and classification of neural action potentials. , 1998, Network.

[19]  P. Kennedy,et al.  Restoration of neural output from a paralyzed patient by a direct brain connection , 1998, Neuroreport.

[20]  Michael G. Lacourse,et al.  Cortical potentials during imagined movements in individuals with chronic spinal cord injuries , 1999, Behavioural Brain Research.

[21]  D. Kipke,et al.  Long-term neural recording characteristics of wire microelectrode arrays implanted in cerebral cortex. , 1999, Brain research. Brain research protocols.

[22]  Miguel A. L. Nicolelis,et al.  Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex , 1999, Nature Neuroscience.

[23]  H. Flor,et al.  A spelling device for the paralysed , 1999, Nature.

[24]  M. Jeannerod,et al.  Mental imaging of motor activity in humans , 1999, Current Opinion in Neurobiology.

[25]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[26]  H. Flor,et al.  The thought translation device (TTD) for completely paralyzed patients. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[27]  P R Kennedy,et al.  Direct control of a computer from the human central nervous system. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[28]  Xiaorong Gao,et al.  Brain-Computer Interface with High Transfer Rates* , 2001 .

[29]  R. Irizarry,et al.  Electrocorticographic gamma activity during word production in spoken and sign language , 2001, Neurology.

[30]  B. Gordon,et al.  Induced electrocorticographic gamma activity during auditory perception , 2001, Clinical Neurophysiology.

[31]  Dawn M. Taylor,et al.  Direct Cortical Control of 3D Neuroprosthetic Devices , 2002, Science.

[32]  Brian D. Ripley,et al.  Modern Applied Statistics with S Fourth edition , 2002 .

[33]  S. Meagher Instant neural control of a movement signal , 2002 .

[34]  David M. Santucci,et al.  Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates , 2003, PLoS biology.

[35]  Jerald D. Kralik,et al.  Chronic, multisite, multielectrode recordings in macaque monkeys , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[36]  Gernot R. Müller-Putz,et al.  "Virtual keyboard" controlled by spontaneous EEG activity , 2001, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[37]  A. Haig,et al.  Impairment, Activity, Participation, Life Satisfaction, and Survival in Persons With Locked‐In Syndrome for Over a Decade: Follow‐Up on a Previously Reported Cohort , 2003, The Journal of head trauma rehabilitation.

[38]  G. R. Muller,et al.  Clinical application of an EEG-based brain–computer interface: a case study in a patient with severe motor impairment , 2003, Clinical Neurophysiology.

[39]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[40]  Coarticulation • Suprasegmentals,et al.  Acoustic Phonetics , 2019, The SAGE Encyclopedia of Human Communication Sciences and Disorders.

[41]  Christa Neuper,et al.  An asynchronously controlled EEG-based virtual keyboard: improvement of the spelling rate , 2004, IEEE Transactions on Biomedical Engineering.

[42]  Andrew B Schwartz,et al.  Cortical neural prosthetics. , 2004, Annual review of neuroscience.

[43]  P.R. Kennedy,et al.  Computer control using human intracortical local field potentials , 2004, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[44]  Jonathan R Wolpaw,et al.  Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[45]  Gerwin Schalk,et al.  A brain–computer interface using electrocorticographic signals in humans , 2004, Journal of neural engineering.

[46]  G. Pfurtscheller,et al.  Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. , 2005, Brain research. Cognitive brain research.

[47]  K. E. Jones,et al.  A glass/silicon composite intracortical electrode array , 2006, Annals of Biomedical Engineering.

[48]  D.J. McFarland,et al.  The wadsworth BCI research and development program: at home with BCI , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[49]  Jon A. Mukand,et al.  Neuronal ensemble control of prosthetic devices by a human with tetraplegia , 2006, Nature.

[50]  Satrajit S. Ghosh,et al.  Neural modeling and imaging of the cortical interactions underlying syllable production , 2006, Brain and Language.

[51]  Tom Chen,et al.  Design and implementation , 2006, IEEE Commun. Mag..

[52]  W. Gormley,et al.  Neuronal ensemble control of prosthetic devices by a human with tetraplegiaHochberg LR, Serruya MD, Friehs GM, et al (Harvard Med School; Brown Univ, Providence, RI; Veterans Health Administration, Providence, RI; et al) Nature 442:164–171, 2006§ , 2007 .

[53]  A. Graser,et al.  Spelling with Steady-State Visual Evoked Potentials , 2007, 2007 3rd International IEEE/EMBS Conference on Neural Engineering.

[54]  Niels Birbaumer,et al.  The thought-translation device , 2007 .

[55]  Michael J. Black,et al.  Assistive technology and robotic control using motor cortex ensemble‐based neural interface systems in humans with tetraplegia , 2007, The Journal of physiology.

[56]  Benjamin Blankertz,et al.  A Note on Brain Actuated Spelling with the Berlin Brain-Computer Interface , 2007, HCI.

[57]  Rajesh P. N. Rao,et al.  Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[58]  P. Kennedy,et al.  Neurotrophic electrode: Method of assembly and implantation into human motor speech cortex , 2008, Journal of Neuroscience Methods.

[59]  Andrew S. Whitford,et al.  Cortical control of a prosthetic arm for self-feeding , 2008, Nature.

[60]  Michael H Kohrman,et al.  ECoG gamma activity during a language task: differentiating expressive and receptive speech areas. , 2008, Brain : a journal of neurology.

[61]  M. Ward,et al.  Toward a comparison of microelectrodes for acute and chronic recordings , 2009, Brain Research.

[62]  F. Guenther,et al.  A Wireless Brain-Machine Interface for Real-Time Speech Synthesis , 2009, PloS one.

[63]  Frank H. Guenther,et al.  Artificial speech synthesizer control by brain-computer interface , 2009, INTERSPEECH.

[64]  Frank H. Guenther,et al.  Brain-computer interfaces for speech communication , 2010, Speech Commun..

[65]  Bradley Greger,et al.  Decoding spoken words using local field potentials recorded from the cortical surface , 2010, Journal of neural engineering.

[66]  J. M. Gilbert,et al.  Silent speech interfaces , 2010, Speech Commun..

[67]  Brett Matthews,et al.  A Probabilistic Decoding Approach to a Neural Prosthesis for Speech , 2010, 2010 4th International Conference on Bioinformatics and Biomedical Engineering.

[68]  William N. Venables,et al.  Modern Applied Statistics with S , 2010 .