evelopment of a hybrid mental spelling system combining SVEP-based brain – computer interface and webcam-based eye racking

Abstract The goal of this study was to develop a hybrid mental speller that can effectively prevent unexpected typing errors in the steady-state visual evoked potential (SSVEP)-based mental speller by simultaneously using the information of eye-gaze direction detected by a low-cost webcam without calibration. In the implemented hybrid mental speller, a character corresponding to the strongest SSVEP response was typed only when the position of the selected character coincided with the horizontal eye-gaze direction (‘left’, ‘no direction’, or ‘right’) detected by the webcam-based eye tracker. When the character detected by the SSVEP-based mental speller was located in the direction opposite the eye-gaze direction, the character was not typed at all (a beep sound was generated instead), and thus the users of the speller did not need to correct the mistyped character using a ‘BACKSPACE’ key. To verify the feasibility and usefulness of the developed hybrid mental spelling system, we conducted online experiments with ten healthy participants, each of whom was asked to type 15 English words consisting of a total of 68 characters. As a result, 16.6 typing errors could be prevented on average, demonstrating that the proposed hybrid strategy could effectively enhance the performance of the SSVEP-based mental spelling system.

[1]  Xiaorong Gao,et al.  Design and implementation of a brain-computer interface with high transfer rates , 2002, IEEE Transactions on Biomedical Engineering.

[2]  R John Leigh,et al.  Eye movements in amyotrophic lateral sclerosis and its mimics: a review with illustrative cases , 2010, Journal of Neurology, Neurosurgery & Psychiatry.

[3]  Tom Chau,et al.  A Brain-Computer Interface Based on Bilateral Transcranial Doppler Ultrasound , 2011, PloS one.

[4]  R. Leigh,et al.  Slow vertical saccades in motor neuron disease: Correlation of structure and function , 1998, Annals of neurology.

[5]  Christa Neuper,et al.  Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic , 2011, Medical & Biological Engineering & Computing.

[6]  Dennis J. McFarland,et al.  The P300-based brain–computer interface (BCI): Effects of stimulus rate , 2011, Clinical Neurophysiology.

[7]  Ying Sun,et al.  Asynchronous P300 BCI: SSVEP-based control state detection , 2010, 2010 18th European Signal Processing Conference.

[8]  Xiaorong Gao,et al.  An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method , 2009, Journal of neural engineering.

[9]  Xiaorong Gao,et al.  A BCI-based environmental controller for the motion-disabled. , 2003, IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[10]  Wolfgang Rosenstiel,et al.  An MEG-based brain–computer interface (BCI) , 2007, NeuroImage.

[11]  Ronald M. Aarts,et al.  A Survey of Stimulation Methods Used in SSVEP-Based BCIs , 2010, Comput. Intell. Neurosci..

[12]  Emmanuel Maby,et al.  Objective and Subjective Evaluation of Online Error Correction during P300-Based Spelling , 2012, Adv. Hum. Comput. Interact..

[13]  Wolfgang Grodd,et al.  Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI) , 2004, IEEE Transactions on Biomedical Engineering.

[14]  E-J Hoogerwerf,et al.  Clinical evaluation of BrainTree, a motor imagery hybrid BCI speller , 2014, Journal of neural engineering.

[15]  A. Cichocki,et al.  Steady-state visually evoked potentials: Focus on essential paradigms and future perspectives , 2010, Progress in Neurobiology.

[16]  Chang-Hwan Im,et al.  EEG-Based Brain-Computer Interfaces: A Thorough Literature Survey , 2013, Int. J. Hum. Comput. Interact..

[17]  Päivi Majaranta,et al.  Twenty years of eye typing: systems and design issues , 2002, ETRA.

[18]  Jonathan R Wolpaw,et al.  A brain-computer interface for long-term independent home use , 2010, Amyotrophic lateral sclerosis : official publication of the World Federation of Neurology Research Group on Motor Neuron Diseases.

[19]  Benjamin Blankertz,et al.  Exploring motion VEPs for gaze-independent communication , 2012, Journal of neural engineering.

[20]  Soo-Young Lee,et al.  Brain–computer interface using fMRI: spatial navigation by thoughts , 2004, Neuroreport.

[21]  Gerwin Schalk,et al.  A brain–computer interface using electrocorticographic signals in humans , 2004, Journal of neural engineering.

[22]  Xun Chen,et al.  Classification of EEG Signals Using a Multiple Kernel Learning Support Vector Machine , 2014, Sensors.

[23]  Tao Liu,et al.  N200-speller using motion-onset visual response , 2009, Clinical Neurophysiology.

[24]  G. Pfurtscheller,et al.  Optimal spatial filtering of single trial EEG during imagined hand movement. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[25]  Chang-Hwan Im,et al.  Development of an SSVEP-based BCI spelling system adopting a QWERTY-style LED keyboard , 2012, Journal of Neuroscience Methods.

[26]  Chern-Sheng Lin,et al.  Eye-controlled virtual keyboard using a new coordinate transformation of long and narrow region , 2008 .

[27]  Hossein Nezamabadi-pour,et al.  Automatic extraction of eye field from a gray intensity image using intensity filtering and hybrid projection function , 2011, 2011 International Conference on Communications, Computing and Control Applications (CCCA).

[28]  J. Wolpaw,et al.  Decoding two-dimensional movement trajectories using electrocorticographic signals in humans , 2007, Journal of neural engineering.

[29]  Hubert Cecotti,et al.  A Self-Paced and Calibration-Less SSVEP-Based Brain–Computer Interface Speller , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[30]  Maureen Clerc,et al.  An analysis of performance evaluation for motor-imagery based BCI , 2013, Journal of neural engineering.

[31]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[32]  Feng Wan,et al.  Single-Trial Detection of Error-Related Potential by One-Unit SOBI-R in SSVEP-Based BCI , 2014, ISNN.

[33]  Dario Farina,et al.  Comparison of feature selection and classification methods for a brain–computer interface driven by non-motor imagery , 2010, Medical & Biological Engineering & Computing.

[34]  Martyn Hill,et al.  Biomedical Signal Processing and Control , 2014 .

[35]  Cuntai Guan,et al.  Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface , 2007, NeuroImage.

[36]  Ivan Volosyak,et al.  SSVEP-based Bremen–BCI interface—boosting information transfer rates , 2011, Journal of neural engineering.

[37]  Shirley M Coyle,et al.  Brain–computer interface using a simplified functional near-infrared spectroscopy system , 2007, Journal of neural engineering.

[38]  Marc M. Van Hulle,et al.  Language Model Applications to Spelling with Brain-Computer Interfaces , 2014, Sensors.

[39]  Tom Chau,et al.  Towards a hemodynamic BCI using transcranial Doppler without user-specific training data. , 2013, Journal of neural engineering.

[40]  B. Okuda,et al.  Motor neuron disease with slow eye movements and vertical gaze palsy , 1992, Acta neurologica Scandinavica.

[41]  Arne Robben,et al.  Error-related potential recorded by EEG in the context of a p300 mind speller brain-computer interface , 2010, 2010 IEEE International Workshop on Machine Learning for Signal Processing.

[42]  Dimitrios Pantazis,et al.  Using Brain Waves to Control Computers and Machines , 2013, Adv. Hum. Comput. Interact..

[43]  G. Pfurtscheller,et al.  EEG-based communication: presence of an error potential , 2000, Clinical Neurophysiology.

[44]  Kyungeun Cho,et al.  A Development Architecture for Serious Games Using BCI (Brain Computer Interface) Sensors , 2012, Sensors.

[45]  Nico M Schmidt,et al.  Online detection of error-related potentials boosts the performance of mental typewriters , 2012, BMC Neuroscience.

[46]  I. Scott MacKenzie,et al.  BlinkWrite: efficient text entry using eye blinks , 2011, Universal Access in the Information Society.

[47]  Xiaorong Gao,et al.  Enhancing the classification accuracy of steady-state visual evoked potential-based brain–computer interfaces using phase constrained canonical correlation analysis , 2011, Journal of neural engineering.

[48]  Xingyu Wang,et al.  A new P300 stimulus presentation pattern for EEG-based spelling systems , 2010, Biomedizinische Technik. Biomedical engineering.

[49]  Wei Wu,et al.  Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs , 2007, IEEE Transactions on Biomedical Engineering.

[50]  Urbano Nunes,et al.  Statistical spatial filtering for a P300-based BCI: Tests in able-bodied, and patients with cerebral palsy and amyotrophic lateral sclerosis , 2011, Journal of Neuroscience Methods.

[51]  Vera Kaiser,et al.  Fast set-up asynchronous brain-switch based on detection of foot motor imagery in 1-channel EEG , 2010, Medical & Biological Engineering & Computing.

[52]  Girijesh Prasad,et al.  Sensorimotor learning with stereo auditory feedback for a brain–computer interface , 2012, Medical & Biological Engineering & Computing.

[53]  Chang-Hwan Im,et al.  Evaluation of feature extraction methods for EEG-based brain–computer interfaces in terms of robustness to slight changes in electrode locations , 2012, Medical & Biological Engineering & Computing.

[54]  A. Cichocki,et al.  Optimization of SSVEP brain responses with application to eight-command Brain–Computer Interface , 2010, Neuroscience Letters.

[55]  Kang Ryoung Park,et al.  Remote Gaze Tracking System on a Large Display , 2013, Sensors.

[56]  Wolfgang Rosenstiel,et al.  Online use of error-related potentials in healthy users and people with severe motor impairment increases performance of a P300-BCI , 2012, Clinical Neurophysiology.

[57]  José del R. Millán,et al.  EEG-Based Brain-Computer Interaction: Improved Accuracy by Automatic Single-Trial Error Detection , 2007, NIPS.

[58]  R. Pieters,et al.  A Review of Eye-Tracking Research in Marketing , 2008 .

[59]  R. Kass,et al.  Decoding and cortical source localization for intended movement direction with MEG. , 2010, Journal of neurophysiology.