Optimising non-invasive brain-computer interface systems for free communication between naïve human participants

Free communication is one of the cornerstones of modern civilisation. While manual keyboards currently allow us to interface with computers and manifest our thoughts, a next frontier is communication without manual input. Brain-computer interface (BCI) spellers often achieve this by decoding patterns of neural activity as users attend to flickering keyboard displays. To date, the highest performing spellers report typing rates of ~10.00 words/minute. While impressive, these rates are typically calculated for experienced users repetitively typing single phrases. It is therefore not clear whether naïve users are able to achieve such high rates with the added cognitive load of genuine free communication, which involves continuously generating and spelling novel words and phrases. In two experiments, we developed an open-source, high-performance, non-invasive BCI speller and examined its feasibility for free communication. The BCI speller required users to focus their visual attention on a flickering keyboard display, thereby producing unique cortical activity patterns for each key, which were decoded using filter-bank canonical correlation analysis. In Experiment 1, we tested whether seventeen naïve users could maintain rapid typing during prompted free word association. We found that information transfer rates were indeed slower during this free communication task than during typing of a cued character sequence. In Experiment 2, we further evaluated the speller’s efficacy for free communication by developing a messaging interface, allowing users to engage in free conversation. The results showed that free communication was possible, but that information transfer was reduced by voluntary textual corrections and turn-taking during conversation. We evaluated a number of factors affecting the suitability of BCI spellers for free communication, and make specific recommendations for improving classification accuracy and usability. Overall, we found that developing a BCI speller for free communication requires a focus on usability over reduced character selection time, and as such, future performance appraisals should be based on genuine free communication scenarios.

[1]  Tzyy-Ping Jung,et al.  High-speed spelling with a noninvasive brain–computer interface , 2015, Proceedings of the National Academy of Sciences.

[2]  Tobias Kaufmann,et al.  Spelling is Just a Click Away – A User-Centered Brain–Computer Interface Including Auto-Calibration and Predictive Text Entry , 2012, Front. Neurosci..

[3]  Xiaorong Gao,et al.  Chirp-modulated visual evoked potential as a generalization of steady state visual evoked potential , 2012, Journal of neural engineering.

[4]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[5]  Matteo Matteucci,et al.  A predictive speller controlled by a brain-computer interface based on motor imagery , 2012, TCHI.

[6]  David J. Ward,et al.  Artificial intelligence: Fast hands-free writing by gaze direction , 2002, Nature.

[7]  Mihaly Benda,et al.  Brain–Computer Interface Spellers: A Review , 2018, Brain sciences.

[8]  Susan Fager,et al.  AAC for adults with acquired neurological conditions: A review , 2007, Augmentative and alternative communication.

[9]  Moritz Grosse-Wentrup,et al.  Multisubject Learning for Common Spatial Patterns in Motor-Imagery BCI , 2011, Comput. Intell. Neurosci..

[10]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces , 2007, Journal of neural engineering.

[11]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[12]  Jussi T Lindgren,et al.  As above, so below? Towards understanding inverse models in BCI , 2018, Journal of neural engineering.

[13]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[14]  Yijun Wang,et al.  A high-speed BCI based on code modulation VEP , 2011, Journal of neural engineering.

[15]  Katsumi Aoki,et al.  Recent development of flow visualization , 2004, J. Vis..

[16]  Takeo Watanabe,et al.  Perceptual learning incepted by decoded fMRI neurofeedback without stimulus presentation , 2012 .

[17]  David Matthews,et al.  Virtual-reality applications give science a new dimension , 2018, Nature.

[18]  Eric W. Sellers,et al.  Predictive Spelling With a P300-Based Brain–Computer Interface: Increasing the Rate of Communication , 2010, Int. J. Hum. Comput. Interact..

[19]  Ramin Skibba,et al.  Virtual reality comes of age , 2018, Nature.

[20]  Nicholas V. Annetta,et al.  Restoring cortical control of functional movement in a human with quadriplegia , 2016, Nature.

[21]  Dean J Krusienski,et al.  A comparison of classification techniques for the P300 Speller , 2006, Journal of neural engineering.

[22]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update , 2018, Journal of neural engineering.

[23]  Robert Oostenveld,et al.  The five percent electrode system for high-resolution EEG and ERP measurements , 2001, Clinical Neurophysiology.

[24]  Francis R. Willett,et al.  High performance communication by people with paralysis using an intracortical brain-computer interface , 2017, eLife.

[25]  Nick Yeung,et al.  The many characters of visual alpha oscillations , 2018, The European journal of neuroscience.

[26]  Tzyy-Ping Jung,et al.  A High-Speed Brain Speller using steady-State Visual evoked potentials , 2014, Int. J. Neural Syst..

[27]  Peng Yuan,et al.  A study of the existing problems of estimating the information transfer rate in online brain–computer interfaces , 2013, Journal of neural engineering.

[28]  W Speier,et al.  Integrating language models into classifiers for BCI communication: a review , 2016, Journal of neural engineering.

[29]  Maureen Clerc,et al.  An analysis of performance evaluation for motor-imagery based BCI , 2013, Journal of neural engineering.

[30]  David J. Ward,et al.  Fast Hands-free Writing by Gaze Direction , 2002, ArXiv.

[31]  Robert C. Wolpert,et al.  A Review of the , 1985 .

[32]  E. W. Sellers,et al.  Toward enhanced P300 speller performance , 2008, Journal of Neuroscience Methods.

[33]  R. Baloh,et al.  Quantitative measurement of saccade amplitude, duration, and velocity , 1975, Neurology.

[34]  Chi Zhang,et al.  A novel P300 BCI speller based on the Triple RSVP paradigm , 2018, Scientific Reports.

[35]  G. Riva,et al.  The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature , 2018, Front. Psychol..

[36]  Maria V. Sanchez-Vives,et al.  From presence to consciousness through virtual reality , 2005, Nature Reviews Neuroscience.

[37]  E. Jones.  The Life and Work of Sigmund Freud , 1964 .

[38]  Clemens Brunner,et al.  BioSig: A Free and Open Source Software Library for BCI Research , 2008, Computer.

[39]  H. Flor,et al.  A spelling device for the paralysed , 1999, Nature.

[40]  Edward Awh,et al.  Spatial attention, preview, and popout: which factors influence critical spacing in crowded displays? , 2007, Journal of vision.

[41]  R Zerafa,et al.  To train or not to train? A survey on training of feature extraction methods for SSVEP-based BCIs , 2018, Journal of neural engineering.

[42]  Xiaogang Chen,et al.  Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface , 2015, Journal of neural engineering.

[43]  Miguel A. L. Nicolelis,et al.  Brain–machine interfaces: past, present and future , 2006, Trends in Neurosciences.

[44]  Wolfgang Rosenstiel,et al.  Online Adaptation of a c-VEP Brain-Computer Interface(BCI) Based on Error-Related Potentials and Unsupervised Learning , 2012, PloS one.

[45]  E. Sellers,et al.  How many people are able to control a P300-based brain–computer interface (BCI)? , 2009, Neuroscience Letters.

[46]  Yijun Wang,et al.  Enhancing Detection of SSVEPs for a High-Speed Brain Speller Using Task-Related Component Analysis , 2018, IEEE Transactions on Biomedical Engineering.

[47]  Nader Pouratian,et al.  A comparison of stimulus types in online classification of the P300 speller using language models , 2017, PloS one.

[48]  Susan L. Travis,et al.  Neural Responses to Target Features outside a Search Array Are Enhanced during Conjunction but Not Unique-Feature Search , 2014, The Journal of Neuroscience.

[49]  Jan Noyes,et al.  The QWERTY Keyboard: A Review , 1983, Int. J. Man Mach. Stud..

[50]  Søren K. Andersen,et al.  Driving steady-state visual evoked potentials at arbitrary frequencies using temporal interpolation of stimulus presentation , 2015, BMC Neuroscience.

[51]  Jason B Mattingley,et al.  Differential Deployment of Visual Attention During Interactive Approach and Avoidance Behavior. , 2019, Cerebral cortex.

[52]  Michael X Cohen,et al.  Where Does EEG Come From and What Does It Mean? , 2017, Trends in Neurosciences.

[53]  Ivan Volosyak,et al.  SSVEP-based Bremen–BCI interface—boosting information transfer rates , 2011, Journal of neural engineering.

[54]  Vladimir V. Bochkarev,et al.  Average word length dynamics as indicator of cultural changes in society , 2012, ArXiv.

[55]  Paul Sajda,et al.  Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials , 2018, Journal of neural engineering.

[56]  Benjamin Blankertz,et al.  A Novel 9-Class Auditory ERP Paradigm Driving a Predictive Text Entry System , 2011, Front. Neurosci..

[57]  Jonathan D. Cohen,et al.  Closed-loop training of attention with real-time brain imaging , 2015, Nature Neuroscience.

[58]  Justin M. Ales,et al.  The steady-state visual evoked potential in vision research: A review. , 2015, Journal of vision.

[59]  Andres Hoyos Idrobo,et al.  Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines , 2016, NeuroImage.

[60]  G Pfurtscheller,et al.  EEG-based communication: improved accuracy by response verification. , 1998, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[61]  Tae-Seong Kim,et al.  A P300-based brain computer interface system for words typing , 2014, Comput. Biol. Medicine.

[62]  Leon Li,et al.  Brain-to-speech decoding will require linguistic and pragmatic data , 2018, Journal of neural engineering.

[63]  Jie Li,et al.  A Synchronous Motor Imagery Based Neural Physiological Paradigm for Brain Computer Interface Speller , 2017, Front. Hum. Neurosci..

[64]  V Bochkarev Vladimir,et al.  The average word length dynamics as an indicator of cultural changes in society , 2015 .

[65]  D. Holdstock.  Past, present--and future? , 2005, Medicine, conflict, and survival.