Toward a reliable gaze-independent hybrid BCI combining visual and natural auditory stimuli

BACKGROUND Brain computer interfaces (BCIs) are one of the last communication options for patients in the locked-in state (LIS). For complete LIS patients, interfaces must be gaze-independent due to their eye impairment. However, unimodal gaze-independent approaches typically present levels of performance substantially lower than gaze-dependent approaches. The combination of multimodal stimuli has been pointed as a viable way to increase users' performance. NEW METHOD A hybrid visual and auditory (HVA) P300-based BCI combining simultaneously visual and auditory stimulation is proposed. Auditory stimuli are based on natural meaningful spoken words, increasing stimuli discrimination and decreasing user's mental effort in associating stimuli to the symbols. The visual part of the interface is covertly controlled ensuring gaze-independency. RESULTS Four conditions were experimentally tested by 10 healthy participants: visual overt (VO), visual covert (VC), auditory (AU) and covert HVA. Average online accuracy for the hybrid approach was 85.3%, which is more than 32% over VC and AU approaches. Questionnaires' results indicate that the HVA approach was the less demanding gaze-independent interface. Interestingly, the P300 grand average for HVA approach coincides with an almost perfect sum of P300 evoked separately by VC and AU tasks. COMPARISON WITH EXISTING METHODS The proposed HVA-BCI is the first solution simultaneously embedding natural spoken words and visual words to provide a communication lexicon. Online accuracy and task demand of the approach compare favorably with state-of-the-art. CONCLUSIONS The proposed approach shows that the simultaneous combination of visual covert control and auditory modalities can effectively improve the performance of gaze-independent BCIs.

[1]  E Donchin,et al.  Brain-computer interface technology: a review of the first international meeting. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[2]  J. Wolpaw,et al.  Clinical Applications of Brain-Computer Interfaces: Current State and Future Prospects , 2009, IEEE Reviews in Biomedical Engineering.

[3]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[4]  Ivo Käthner,et al.  Comparison of eye tracking, electrooculography and an auditory brain-computer interface for binary communication: a case study with a participant in the locked-in state , 2015, Journal of NeuroEngineering and Rehabilitation.

[5]  J. Wolpaw,et al.  A practical, intuitive brain–computer interface for communicating ‘yes’ or ‘no’ by listening , 2014, Journal of neural engineering.

[6]  Jonathan R Wolpaw,et al.  A brain-computer interface for long-term independent home use , 2010, Amyotrophic lateral sclerosis : official publication of the World Federation of Neurology Research Group on Motor Neuron Diseases.

[7]  S. Debener,et al.  Towards a truly mobile auditory brain-computer interface: exploring the P300 to take away. , 2014, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[8]  Steven Laureys,et al.  A Comparison of Two Spelling Brain-Computer Interfaces Based on Visual P3 and SSVEP in Locked-In Syndrome , 2013, PloS one.

[9]  Donatella Mattia,et al.  User-centred design in brain-computer interface research and development. , 2015, Annals of physical and rehabilitation medicine.

[10]  A. Kübler,et al.  A Brain–Computer Interface Controlled Auditory Event‐Related Potential (P300) Spelling System for Locked‐In Patients , 2009, Annals of the New York Academy of Sciences.

[11]  F. Gerstenbrand,et al.  Varieties of the locked-in syndrome , 1979, Journal of Neurology.

[12]  Tomasz M. Rutkowski,et al.  Tactile and bone-conduction auditory brain computer interface for vision and hearing impaired users , 2014, Journal of Neuroscience Methods.

[13]  P. Hagoort.  The fractionation of spoken language understanding by measuring electrical and magnetic brain signals , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[14]  Niels Birbaumer,et al.  Brain–computer-interface research: Coming of age , 2006, Clinical Neurophysiology.

[15]  J. Wolpaw,et al.  Answering questions with an electroencephalogram-based brain-computer interface. , 1998, Archives of physical medicine and rehabilitation.

[16]  J. Wolpaw,et al.  A P300-based brain–computer interface for people with amyotrophic lateral sclerosis , 2008, Clinical Neurophysiology.

[17]  B. Blankertz,et al.  (C)overt attention and visual speller design in an ERP-based brain-computer interface , 2010, Behavioral and Brain Functions.

[18]  Xiaorong Gao,et al.  An auditory BCI using voluntary mental response , 2009, 2009 4th International IEEE/EMBS Conference on Neural Engineering.

[19]  Robert Riener,et al.  Control strategies for active lower extremity prosthetics and orthotics: a review , 2015, Journal of NeuroEngineering and Rehabilitation.

[20]  N. Suga,et al.  Auditory System , 2020, Definitions.

[21]  Kouji Takano,et al.  A region-based two-step P300-based brain–computer interface for patients with amyotrophic lateral sclerosis , 2014, Clinical Neurophysiology.

[22]  Jonathan R Wolpaw,et al.  Sensorimotor rhythm-based brain–computer interface (BCI): model order selection for autoregressive spectral analysis , 2008, Journal of neural engineering.

[23]  Gerwin Schalk,et al.  THE P300 AS A TYPING TOOL: TESTS OF BRAIN COMPUTER INTERFACE WITH AN ALS PATIENT , 2000 .

[24]  Benjamin Blankertz,et al.  A Novel 9-Class Auditory ERP Paradigm Driving a Predictive Text Entry System , 2011, Front. Neurosci..

[25]  Ivo Käthner,et al.  Rapid P300 brain-computer interface communication with a head-mounted display , 2015, Front. Neurosci..

[26]  M. Ermani,et al.  Survival and quality of life after tracheostomy for acute respiratory failure in patients with amyotrophic lateral sclerosis. , 2011, Journal of critical care.

[27]  Urbano Nunes,et al.  GIBS block speller: Toward a gaze-independent P300-based BCI , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[28]  Rachael D. Seidler,et al.  A simple solution for model comparison in bold imaging: the special case of reward prediction error and reward outcomes , 2013, Front. Neurosci..

[29]  M. Castelo‐Branco,et al.  Comparison of a row-column speller vs. a novel lateral single-character speller: Assessment of BCI for severe motor disabled patients , 2012, Clinical Neurophysiology.

[30]  Makoto Wada,et al.  Effect of the Green/Blue Flicker Matrix for P300-Based Brain–Computer Interface: An EEG–fMRI Study , 2012, Front. Neur..

[31]  Atsushi Maki,et al.  High cognitive function of an ALS patient in the totally locked-in state , 2008, Neuroscience Letters.

[32]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[33]  N. Birbaumer,et al.  An auditory oddball brain–computer interface for binary choices , 2010, Clinical Neurophysiology.

[34]  Tobias Kaufmann,et al.  Comparison of tactile, auditory, and visual modality for brain-computer interface use: a case study with a patient in the locked-in state , 2013, Front. Neurosci..

[35]  Urbano Nunes,et al.  Statistical spatial filtering for a P300-based BCI: Tests in able-bodied, and patients with cerebral palsy and amyotrophic lateral sclerosis , 2011, Journal of Neuroscience Methods.

[36]  Steven Laureys,et al.  The locked-in syndrome : what is it like to be conscious but paralyzed and voiceless? , 2005, Progress in brain research.

[37]  Dong Ming,et al.  Exploring Combinations of Auditory and Visual Stimuli for Gaze-Independent Brain-Computer Interfaces , 2014, PloS one.

[38]  Brendan Z Allison,et al.  Effects of SOA and flash pattern manipulations on ERPs, performance, and preference: implications for a BCI system. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[39]  N. Birbaumer,et al.  An auditory oddball (P300) spelling system for brain-computer interfaces. , 2009, Psychophysiology.

[40]  M. Molinari,et al.  Brain–computer interface boosts motor imagery practice during stroke recovery , 2015, Annals of neurology.

[41]  Urbano Nunes,et al.  Assisted navigation for a brain-actuated intelligent wheelchair , 2013, Robotics Auton. Syst..

[42]  E. Donchin,et al.  A P300-based brain–computer interface: Initial tests by ALS patients , 2006, Clinical Neurophysiology.

[43]  N. Birbaumer,et al.  Prediction of Auditory and Visual P300 Brain-Computer Interface Aptitude , 2013, PloS one.

[44]  Gernot R. Müller-Putz,et al.  The auditory P300-based single-switch brain-computer interface: Paradigm transition from healthy subjects to minimally conscious patients , 2013, Artif. Intell. Medicine.

[45]  Michael Tangermann,et al.  Natural stimuli improve auditory BCIs with respect to ergonomics and performance. , 2012, Journal of neural engineering.

[46]  Christoph Braun,et al.  A portable auditory P300 brain–computer interface with directional cues , 2013, Clinical Neurophysiology.

[47]  M. Tervaniemi,et al.  Lateralization of auditory-cortex functions , 2003, Brain Research Reviews.

[48]  G Müller-Putz,et al.  An independent SSVEP-based brain–computer interface in locked-in syndrome , 2014, Journal of neural engineering.

[49]  Gernot R. Müller-Putz,et al.  Effects of mental workload and fatigue on the P300, alpha and theta band power during operation of an ERP (P300) brain–computer interface , 2014, Biological Psychology.

[50]  B. Schoelkopf,et al.  Transition from the locked in to the completely locked-in state: A physiological analysis , 2011, Clinical Neurophysiology.

[51]  Andrzej Cichocki,et al.  Hybrid-BCI: Classification of auditory and visual related potentials , 2014, 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS) and 15th International Symposium on Advanced Intelligent Systems (ISIS).

[52]  N. Birbaumer,et al.  Brain–computer interfaces and communication in paralysis: Extinction of goal directed thinking in completely paralysed patients? , 2008, Clinical Neurophysiology.

[53]  Robert R. Capranica,et al.  9- – THE AUDITORY SYSTEM , 1976 .

[54]  B. Schölkopf,et al.  An online brain-computer interface based on shifting attention to concurrent streams of auditory stimuli. , 2012, Journal of neural engineering.

[55]  Bernhard Schölkopf,et al.  An Auditory Paradigm for Brain-Computer Interfaces , 2004, NIPS.

[56]  B. Blankertz,et al.  A New Auditory Multi-Class Brain-Computer Interface Paradigm: Spatial Hearing as an Informative Cue , 2010, PloS one.

[57]  E. Hoster,et al.  Diagnostic and ethical challenges in disorders of consciousness and locked-in syndrome: a survey of German neurologists , 2012, Journal of Neurology.

[58]  Cathy J. Price,et al.  Explaining Left Lateralization for Words in the Ventral Occipitotemporal Cortex , 2011, The Journal of Neuroscience.

[59]  Niels Birbaumer,et al.  Ideomotor silence: the case of complete paralysis and brain–computer interfaces (BCI) , 2012, Psychological Research.

[60]  J. Farquhar,et al.  Comparing tactile and visual gaze-independent brain–computer interfaces in patients with amyotrophic lateral sclerosis and healthy users , 2014, Clinical Neurophysiology.

[61]  F. Cincotti,et al.  Eye-gaze independent EEG-based brain–computer interfaces for communication , 2012, Journal of neural engineering.