Meeting brain–computer interface user performance expectations using a deep neural network decoding framework

Brain–computer interface (BCI) neurotechnology has the potential to reduce disability associated with paralysis by translating neural activity into control of assistive devices1–9. Surveys of potential end-users have identified key BCI system features10–14, including high accuracy, minimal daily setup, rapid response times, and multifunctionality. These performance characteristics are primarily influenced by the BCI’s neural decoding algorithm1,15, which is trained to associate neural activation patterns with intended user actions. Here, we introduce a new deep neural network16 decoding framework for BCI systems enabling discrete movements that addresses these four key performance characteristics. Using intracortical data from a participant with tetraplegia, we provide offline results demonstrating that our decoder is highly accurate, sustains this performance beyond a year without explicit daily retraining by combining it with an unsupervised updating procedure3,17–20, responds faster than competing methods8, and can increase functionality with minimal retraining by using a technique known as transfer learning21. We then show that our participant can use the decoder in real-time to reanimate his paralyzed forearm with functional electrical stimulation (FES), enabling accurate manipulation of three objects from the grasp and release test (GRT)22. These results demonstrate that deep neural network decoders can advance the clinical translation of BCI technology.Intracortical activity data recorded over 2 years in a tetraplegic patient is used to develop an artificial intelligence algorithm that achieves fast, accurate, and stable movement decoding to reenable real-time control of the paralyzed forearm.

[1]  G. R. Muller,et al.  Brain oscillations control hand orthosis in a tetraplegic , 2000, Neuroscience Letters.

[2]  Senén Barro,et al.  Do we need hundreds of classifiers to solve real world classification problems? , 2014, J. Mach. Learn. Res..

[3]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[4]  Nicolas Y. Masse,et al.  Advantages of closed-loop calibration in intracortical brain–computer interfaces for people with tetraplegia , 2013, Journal of neural engineering.

[5]  Martial Hebert,et al.  Semi-Supervised Self-Training of Object Detection Models , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[6]  Robert Ajemian,et al.  Neurosurgery: Gentler alternatives to chips in the brain. , 2017, Nature.

[7]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[8]  Nicolas Y. Masse,et al.  Reach and grasp by people with tetraplegia using a neurally controlled robotic arm , 2012, Nature.

[9]  W. Cleveland,et al.  Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting , 1988 .

[10]  J. Huggins,et al.  What would brain-computer interface users want: opinions and priorities of potential users with spinal cord injury. , 2015, Archives of physical medicine and rehabilitation.

[11]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[12]  Mingming Zhang,et al.  Big data challenges in decoding cortical activity in a human with quadriplegia to inform a brain computer interface , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[13]  K. Anderson Targeting recovery: priorities of the spinal cord-injured population. , 2004, Journal of neurotrauma.

[14]  Gerhard Friehs,et al.  Intra-day signal instabilities affect decoding performance in an intracortical neural interface system , 2013, Journal of neural engineering.

[15]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[16]  Giacomo Valle,et al.  The Connection Between the Nervous System and Machines: Commentary , 2019, Journal of medical Internet research.

[17]  M. Keith,et al.  Development of a quantitative hand grasp and release test for patients with tetraplegia using a hand neuroprosthesis. , 1994, The Journal of hand surgery.

[18]  Anish A. Sarma,et al.  Clinical translation of a high-performance neural prosthesis , 2015, Nature Medicine.

[19]  R. Andersen,et al.  Decoding motor imagery from the posterior parietal cortex of a tetraplegic human , 2015, Science.

[20]  M L Boninger,et al.  Ten-dimensional anthropomorphic arm control in a human brain−machine interface: difficulties, solutions, and limitations , 2015, Journal of neural engineering.

[21]  David A. Friedenberg,et al.  Using an Artificial Neural Bypass to Restore Cortical Control of Rhythmic Movements in a Human with Quadriplegia , 2016, Scientific Reports.

[22]  Paul Nuyujukian,et al.  Performance sustaining intracortical neural prostheses , 2014, Journal of neural engineering.

[23]  A. Schwartz,et al.  High-performance neuroprosthetic control by an individual with tetraplegia , 2013, The Lancet.

[24]  N. Birbaumer,et al.  Brain–computer interfaces for communication and rehabilitation , 2016, Nature Reviews Neurology.

[25]  Robert D Flint,et al.  Long-Term Stability of Motor Cortical Activity: Implications for Brain Machine Interfaces and Optimal Feedback Control , 2016, The Journal of Neuroscience.

[26]  Dumitru Erhan,et al.  Training Deep Neural Networks on Noisy Labels with Bootstrapping , 2014, ICLR.

[27]  J Esnouf,et al.  The functional impact of the Freehand System on tetraplegic hand function. Clinical Results , 2002, Spinal Cord.

[28]  Francis R. Willett,et al.  High performance communication by people with paralysis using an intracortical brain-computer interface , 2017, eLife.

[29]  Steven Laureys,et al.  Enhancing clinical communication assessments using an audiovisual BCI for patients with disorders of consciousness , 2017, Journal of neural engineering.

[30]  Gaurav Sharma,et al.  Big data challenges in decoding cortical activity in a human with quadriplegia to inform a brain computer interface. , 2016, Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference.

[31]  Michael L Boninger,et al.  Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury. , 2013, Journal of rehabilitation research and development.

[32]  G. Pfurtscheller,et al.  EEG-based neuroprosthesis control: A step towards clinical practice , 2005, Neuroscience Letters.

[33]  J. Huggins,et al.  What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis , 2011, Amyotrophic lateral sclerosis : official publication of the World Federation of Neurology Research Group on Motor Neuron Diseases.

[34]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[35]  Francis R. Willett,et al.  Restoration of reaching and grasping in a person with tetraplegia through brain-controlled muscle stimulation: a proof-of-concept demonstration , 2017, The Lancet.

[36]  Konrad P. Körding,et al.  Machine Learning for Neural Decoding , 2017, eNeuro.

[37]  P. H. Peckham,et al.  Controlled prehension and release in the C5 quadriplegic elicited by functional electrical stimulation of the paralyzed forearm musculature , 2006, Annals of Biomedical Engineering.

[38]  Michael J. Black,et al.  Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array , 2011 .

[39]  David A. Friedenberg,et al.  Moving a Paralyzed Hand—A Biomedical Big Data Success Story , 2016 .

[40]  Nicolas Y. Masse,et al.  Neural Point-and-Click Communication by a Person With Incomplete Locked-In Syndrome , 2015, Neurorehabilitation and neural repair.

[41]  Leigh R. Hochberg,et al.  Retrospectively supervised click decoder calibration for self-calibrating point-and-click brain–computer interfaces , 2016, Journal of Physiology-Paris.

[42]  Nicholas V. Annetta,et al.  Restoring cortical control of functional movement in a human with quadriplegia , 2016, Nature.

[43]  Nicolas Y. Masse,et al.  Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface , 2015, Science Translational Medicine.

[44]  David Sussillo,et al.  Making brain–machine interfaces robust to future neural variability , 2016, Nature Communications.

[45]  David A. Friedenberg,et al.  Neuroprosthetic-enabled control of graded arm muscle contraction in a paralyzed human , 2017, Scientific Reports.

[46]  David A. Friedenberg,et al.  Time Stability and Coherence Analysis of Multiunit, Single-Unit and Local Field Potential Neuronal Signals in Chronically Implanted Brain Electrodes , 2015 .

[47]  Byron M. Yu,et al.  Self-recalibrating classifiers for intracortical brain–computer interfaces , 2014, Journal of neural engineering.

[48]  Krishna V. Shenoy,et al.  Information Systems Opportunities in Brain–Machine Interface Decoders , 2014, Proceedings of the IEEE.

[49]  Yong Fan,et al.  Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networks , 2019, NeuroImage.

[50]  M. Nicolelis,et al.  Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation. , 2017, Physiological reviews.

[51]  Miguel A. L. Nicolelis,et al.  Adaptive Decoding for Brain-Machine Interfaces Through Bayesian Parameter Updates , 2011, Neural Computation.

[52]  Maarten J. IJzerman,et al.  Survey of the needs of patients with spinal cord injury: impact and priority for improvement in hand function in tetraplegics , 2004, Spinal Cord.

[53]  Mingming Zhang,et al.  Dexterous Control of Seven Functional Hand Movements Using Cortically-Controlled Transcutaneous Muscle Stimulation in a Person With Tetraplegia , 2018, Front. Neurosci..

[54]  Olaf Blanke,et al.  Visual Feedback Dominates the Sense of Agency for Brain-Machine Actions , 2015, PloS one.