A High Performance Spelling System based on EEG-EOG Signals With Visual Feedback

In this paper, we propose a highly accurate and fast spelling system that employs multi-modal electroencephalography-electrooculography (EEG-EOG) signals and visual feedback technology. Over the last 20 years, various types of speller systems have been developed in brain-computer interface and EOG/eye-tracking research; however, these conventional systems have a tradeoff between the spelling accuracy (or decoding) and typing speed. Healthy users and physically challenged participants, in particular, may become exhausted quickly; thus, there is a need for a speller system with fast typing speed while retaining a high level of spelling accuracy. In this paper, we propose the first hybrid speller system that combines EEG and EOG signals with visual feedback technology so that the user and the speller system can act cooperatively for optimal decision-making. The proposed spelling system consists of a classic row-column event-related potential (ERP) speller, an EOG command detector, and visual feedback modules. First, the online ERP speller calculates classification probabilities for all candidate characters from the EEG epochs. Second, characters are sorted by their probability, and the characters with the highest probabilities are highlighted as visual feedback within the row-column spelling layout. Finally, the user can actively select the character as the target by generating an EOG command. The proposed system shows 97.6% spelling accuracy and an information transfer rate of 39.6 (±13.2) [bits/min] across 20 participants. In our extended experiment, we redesigned the visual feedback and minimized the number of channels (four channels) in order to enhance the speller performance and increase usability. Most importantly, a new weighted strategy resulted in 100% accuracy and a 57.8 (±23.6) [bits/min] information transfer rate across six participants. This paper demonstrates that the proposed system can provide a reliable communication channel for practical speller applications and may be used to supplement existing systems.

[1]  Shuai Liu,et al.  Use of a Green Familiar Faces Paradigm Improves P300-Speller Brain-Computer Interface Performance , 2015, PloS one.

[2]  Cuntai Guan,et al.  P300 Brain-Computer Interface Design for Communication and Control Applications , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[3]  Leslie M. Collins,et al.  Utilizing a Language Model to Improve Online Dynamic Data Collection in P300 Spellers , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[4]  Klaus-Robert Müller,et al.  Classifying Single Trial EEG: Towards Brain Computer Interfacing , 2001, NIPS.

[5]  N. Birbaumer,et al.  Brain–computer interfaces and communication in paralysis: Extinction of goal directed thinking in completely paralysed patients? , 2008, Clinical Neurophysiology.

[6]  Chang-Hwan Im,et al.  Development of a hybrid mental speller combining EEG-based brain-computer interface and webcam-based eye-tracking , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[7]  Stefan Haufe,et al.  Single-trial analysis and classification of ERP components — A tutorial , 2011, NeuroImage.

[8]  M. Arns,et al.  Efficacy of Neurofeedback Treatment in ADHD: The Effects on Inattention, Impulsivity and Hyperactivity: A Meta-Analysis , 2009, Clinical EEG and neuroscience.

[9]  W. A. Sarnacki,et al.  Brain–computer interface (BCI) operation: optimizing information transfer rates , 2003, Biological Psychology.

[10]  Mario Cortese,et al.  Hybrid EEG/EOG-based brain/neural hand exoskeleton restores fully independent daily living activities after quadriplegia , 2016, Science Robotics.

[11]  G. Vecchiato,et al.  A hybrid platform based on EOG and EEG signals to restore communication for patients afflicted with progressive motor neuron diseases , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[12]  Dong Ming,et al.  A hybrid BCI speller paradigm combining P300 potential and the SSVEP blocking feature , 2013, Journal of neural engineering.

[13]  Xiaogang Chen,et al.  An online hybrid BCI system based on SSVEP and EMG , 2016, Journal of neural engineering.

[14]  Nader Pouratian,et al.  Integrating Language Information With a Hidden Markov Model to Improve Communication Rate in the P300 Speller , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[15]  Luca T. Mainardi,et al.  Online Detection of P300 and Error Potentials in a BCI Speller , 2010, Comput. Intell. Neurosci..

[16]  F. L. D. Silva,et al.  Event-related EEG/MEG synchronization and desynchronization: basic principles , 1999, Clinical Neurophysiology.

[17]  Y. Anzai,et al.  Pattern Recognition & Machine Learning , 2016 .

[18]  A. Lenhardt,et al.  An Adaptive P300-Based Online Brain–Computer Interface , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[19]  Yuanqing Li,et al.  A Single-Channel EOG-Based Speller , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[20]  Yuanqing Li,et al.  An asynchronous wheelchair control by hybrid EEG–EOG brain–computer interface , 2014, Cognitive Neurodynamics.

[21]  G.F. Inbar,et al.  An improved P300-based brain-computer interface , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[22]  Seong-Whan Lee,et al.  A brain-computer interface speller using peripheral stimulus-based SSVEP and P300 , 2017, 2017 5th International Winter Conference on Brain-Computer Interface (BCI).

[23]  Rami Saab,et al.  A Hybrid Brain–Computer Interface Based on the Fusion of P300 and SSVEP Scores , 2015, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[24]  Benjamin Blankertz,et al.  A novel brain-computer interface based on the rapid serial visual presentation paradigm , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[25]  Seungjin Choi,et al.  A hybrid EOG-P300 BCI with dual monitors , 2014, 2014 International Winter Workshop on Brain-Computer Interface (BCI).

[26]  Siamac Fazli,et al.  Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI , 2015, Pattern Recognit..

[27]  K. A. Colwell,et al.  Bayesian Approach to Dynamically Controlling Data Collection in P300 Spellers , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[28]  A. Kübler,et al.  Face stimuli effectively prevent brain–computer interface inefficiency in patients with neurodegenerative disease , 2013, Clinical Neurophysiology.

[29]  Chun-Liang Hsu,et al.  EOG-based Human-Computer Interface system development , 2010, Expert Syst. Appl..

[30]  Doru Talaba,et al.  P300-Based Brain-Neuronal Computer Interaction for Spelling Applications , 2013, IEEE Transactions on Biomedical Engineering.

[31]  M S Treder,et al.  Gaze-independent brain–computer interfaces based on covert attention and feature attention , 2011, Journal of neural engineering.

[32]  Elisabeth André,et al.  EMG-based hand gesture recognition for realtime biosignal interfacing , 2008, IUI '08.

[33]  Yuanqing Li,et al.  Target Selection With Hybrid Feature for BCI-Based 2-D Cursor Control , 2012, IEEE Transactions on Biomedical Engineering.

[34]  F. Gerstenbrand,et al.  Varieties of the locked-in syndrome , 1979, Journal of Neurology.

[35]  Päivi Majaranta,et al.  Twenty years of eye typing: systems and design issues , 2002, ETRA.

[36]  Brendan Z Allison,et al.  Effects of SOA and flash pattern manipulations on ERPs, performance, and preference: implications for a BCI system. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[37]  Eric W. Sellers,et al.  A general P300 brain–computer interface presentation paradigm based on performance guided constraints , 2012, Neuroscience Letters.

[38]  Serkan Gurkan,et al.  Design of a Novel Efficient Human–Computer Interface: An Electrooculagram Based Virtual Keyboard , 2010, IEEE Transactions on Instrumentation and Measurement.

[39]  T W Picton,et al.  The P300 Wave of the Human Event‐Related Potential , 1992, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[40]  Yodchanan Wongsawat,et al.  Navigation-synchronized multimodal control wheelchair from brain to alternative assistive technologies for persons with severe disabilities , 2017, Cognitive Neurodynamics.

[41]  Dewen Hu,et al.  Hybrid Brain-Computer Interface (BCI) based on the EEG and EOG signals. , 2014, Bio-medical materials and engineering.

[42]  M Adjouadi,et al.  A practical EMG-based human-computer interface for users with motor disabilities. , 2000, Journal of rehabilitation research and development.

[43]  Klaus-Robert Müller,et al.  Introduction to machine learning for brain imaging , 2011, NeuroImage.

[44]  Yoshiaki Hayashi,et al.  Towards Hybrid EEG-EMG-Based Control Approaches to be Used in Bio-robotics Applications: Current Status, Challenges and Future Directions , 2013, Paladyn J. Behav. Robotics.

[45]  Steven L. Johnson,et al.  A P300-Based Brain–Computer Interface: Effects of Interface Type and Screen Size , 2010, Int. J. Hum. Comput. Interact..

[46]  Brendan Z. Allison,et al.  The Hybrid BCI , 2010, Frontiers in Neuroscience.

[47]  J. Ushiba,et al.  Effects of neurofeedback training with an electroencephalogram-based brain-computer interface for hand paralysis in patients with chronic stroke: a preliminary case series study. , 2011, Journal of rehabilitation medicine.

[48]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[49]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[50]  M. Mazo,et al.  System for assisted mobility using eye movements based on electrooculography , 2002, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[51]  K.-R. Muller,et al.  Linear and nonlinear methods for brain-computer interfaces , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[52]  Fumitoshi Matsuno,et al.  A Novel EOG/EEG Hybrid Human–Machine Interface Adopting Eye Movements and ERPs: Application to Robot Control , 2015, IEEE Transactions on Biomedical Engineering.

[53]  Thomas Baer,et al.  Multi‐channel study of tongue EMG during vowel production , 1982 .

[54]  Seong-Whan Lee,et al.  Quantifying movement intentions with multimodal neuroimaging for functional electrical stimulation-based rehabilitation , 2016, Neuroreport.

[55]  Ji-Hoon Jeong,et al.  OpenBMI: A real-time data analysis toolbox for Brain-Machine Interfaces , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[56]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[57]  Manuel Mazo,et al.  Wheelchair Guidance Strategies Using EOG , 2002, J. Intell. Robotic Syst..

[58]  Zhao Lv,et al.  A novel eye movement detection algorithm for EOG driven human computer interface , 2010, Pattern Recognit. Lett..

[59]  Klaus-Robert Müller,et al.  An Efficient ERP-Based Brain-Computer Interface Using Random Set Presentation and Face Familiarity , 2014, PloS one.

[60]  Gerhard Tröster,et al.  Eye Movement Analysis for Activity Recognition Using Electrooculography , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Siamac Fazli,et al.  Development of an open source platform for brain-machine interface: openBMI , 2016, 2016 4th International Winter Conference on Brain-Computer Interface (BCI).

[62]  Yuanqing Li,et al.  An EOG-Based Human–Machine Interface for Wheelchair Control , 2018, IEEE Transactions on Biomedical Engineering.

[63]  A. Cichocki,et al.  The Changing Face of P300 BCIs: A Comparison of Stimulus Changes in a P300 BCI Involving Faces, Emotion, and Movement , 2012, PloS one.

[64]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[65]  I. Scott MacKenzie,et al.  Effects of feedback and dwell time on eye typing speed and accuracy , 2006, Universal Access in the Information Society.

[66]  Xingyu Wang,et al.  Targeting an efficient target-to-target interval for P300 speller brain–computer interfaces , 2012, Medical & Biological Engineering & Computing.

[67]  Stephen A. Brewster,et al.  Foot tapping for mobile interaction , 2010, BCS HCI.

[68]  Yupeng Wang,et al.  A dynamic submatrix-based P300 online brain-computer interface , 2015, Biomed. Signal Process. Control..

[69]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.