An Auditory-Tactile Visual Saccade-Independent P300 Brain-Computer Interface

Most P300 event-related potential (ERP)-based brain-computer interface (BCI) studies focus on gaze shift-dependent BCIs, which cannot be used by people who have lost voluntary eye movement. However, the performance of visual saccade-independent P300 BCIs is generally poor. To improve saccade-independent BCI performance, we propose a bimodal P300 BCI approach that simultaneously employs auditory and tactile stimuli. The proposed P300 BCI is a vision-independent system because no visual interaction is required of the user. Specifically, we designed a direction-congruent bimodal paradigm by randomly and simultaneously presenting auditory and tactile stimuli from the same direction. Furthermore, the channels and number of trials were tailored to each user to improve online performance. With 12 participants, the average online information transfer rate (ITR) of the bimodal approach improved by 45.43% and 51.05% over that attained, respectively, with the auditory and tactile approaches individually. Importantly, the average online ITR of the bimodal approach, including the break time between selections, reached 10.77 bits/min. These findings suggest that the proposed bimodal system holds promise as a practical visual saccade-independent P300 BCI.

[1]  Tapani Ristaniemi,et al.  Low-rank Approximation Based non-Negative Multi-Way Array Decomposition on Event-Related potentials , 2014, Int. J. Neural Syst..

[2]  J. W. Minett,et al.  Optimizing the P300-based brain–computer interface: current status, limitations and future directions , 2011, Journal of neural engineering.

[3]  Bjørn Sætrevik,et al.  The right ear advantage revisited: Speech lateralisation in dichotic listening using consonant–vowel and vowel–consonant syllables , 2012, Laterality.

[4]  M S Treder,et al.  Gaze-independent brain–computer interfaces based on covert attention and feature attention , 2011, Journal of neural engineering.

[5]  Bernhard Schölkopf,et al.  An Auditory Paradigm for Brain-Computer Interfaces , 2004, NIPS.

[6]  S. Bozinovski,et al.  Mobile Robot Trajectory Control: From Fixed Rails to Direct Bioelectric Control , 1990, Proceedings of the IEEE International Workshop on Intelligent Motion Control.

[7]  N Birbaumer,et al.  Decoding of motor intentions from epidural ECoG recordings in severely paralyzed chronic stroke patients , 2014, Journal of neural engineering.

[8]  David J. Ostry,et al.  Temporal factors affecting somatosensory–auditory interactions in speech processing , 2014, Front. Psychol..

[9]  Takayuki Nishimura,et al.  Effect of empathy trait on attention to various facial expressions: evidence from N170 and late positive potential (LPP) , 2014, Journal of Physiological Anthropology.

[10]  K. Lafleur,et al.  Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface , 2013, Journal of neural engineering.

[11]  L. Trainor,et al.  Effects of spatial separation and stimulus probability on the event-related potentials elicited by occasional changes in sound location , 2006, Brain Research.

[12]  Benjamin Blankertz,et al.  Does bimodal stimulus presentation increase ERP components usable in BCIs? , 2012, Journal of neural engineering.

[13]  T. Chau,et al.  A Review of EEG-Based Brain-Computer Interfaces as Access Pathways for Individuals with Severe Disabilities , 2013, Assistive technology : the official journal of RESNA.

[14]  H. Adeli,et al.  Brain-computer interface technologies: from signal to action , 2013, Reviews in the neurosciences.

[15]  E. John,et al.  Evoked-Potential Correlates of Stimulus Uncertainty , 1965, Science.

[16]  Charles Spence,et al.  Audiotactile interactions in front and rear space , 2011, Neuroscience & Biobehavioral Reviews.

[17]  Benjamin Blankertz,et al.  Gaze-independent BCI-spelling using rapid serial visual presentation (RSVP) , 2013, Clinical Neurophysiology.

[18]  J.J. Vidal,et al.  Real-time detection of brain events in EEG , 1977, Proceedings of the IEEE.

[19]  Tzyy-Ping Jung,et al.  A High-Speed Brain Speller using steady-State Visual evoked potentials , 2014, Int. J. Neural Syst..

[20]  Alessandro Farnè,et al.  Auditory peripersonal space in humans. , 2002, Journal of cognitive neuroscience.

[21]  Xingyu Wang,et al.  Aggregation of Sparse Linear Discriminant analyses for Event-Related potential Classification in Brain-Computer Interface , 2014, Int. J. Neural Syst..

[22]  M. Ding,et al.  Attentional modulation of the somatosensory mu rhythm , 2011, Neuroscience.

[23]  Stevo Bozinovski.  Controlling Robots Using EEG Signals, Since 1988 , 2012, ICT Innovations.

[24]  Marieke E. Thurlings,et al.  Controlling a Tactile ERP–BCI in a Dual Task , 2013, IEEE Transactions on Computational Intelligence and AI in Games.

[25]  A Belitski,et al.  P300 audio-visual speller , 2011, Journal of neural engineering.

[26]  Peter Desain,et al.  Introducing the tactile speller: an ERP-based brain–computer interface for communication , 2012, Journal of neural engineering.

[27]  Francisco J. Pelayo,et al.  An auditory Brain-Computer Interface with Accuracy Prediction , 2012, Int. J. Neural Syst..

[28]  Stevo Bozinovski,et al.  Electroexpectogram: Experimental Design and Agorithms , 1992, Proceedings of the 1992 International Biomedical Engineering Days.

[29]  E Donchin,et al.  Brain-computer interface technology: a review of the first international meeting. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[30]  Jan B. F. van Erp,et al.  A Tactile P300 Brain-Computer Interface , 2010, Front. Neurosci..

[31]  J. Wolpaw,et al.  A novel P300-based brain–computer interface stimulus presentation paradigm: Moving beyond rows and columns , 2010, Clinical Neurophysiology.

[32]  Marieke E. Thurlings,et al.  Gaze-independent ERP-BCIs: augmenting performance through location-congruent bimodal stimuli , 2014, Front. Syst. Neurosci..

[33]  Jonathan R Wolpaw,et al.  Brain–computer interface systems: progress and prospects , 2007, Expert review of medical devices.

[34]  E. Donchin,et al.  A P300-based brain–computer interface: Initial tests by ALS patients , 2006, Clinical Neurophysiology.

[35]  Minpeng Xu,et al.  Channel Selection Based on Phase Measurement in P300-Based Brain-Computer Interface , 2013, PloS one.

[36]  K. A. Colwell,et al.  Channel selection methods for the P300 Speller , 2014, Journal of Neuroscience Methods.

[37]  Michael Tangermann,et al.  Towards User-Friendly Spelling with an Auditory Brain-Computer Interface: The CharStreamer Paradigm , 2014, PloS one.

[38]  Brice Rebsamen,et al.  A brain controlled wheelchair to navigate in familiar environments. , 2010, IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[39]  Dong Ming,et al.  Exploring Combinations of Auditory and Visual Stimuli for Gaze-Independent Brain-Computer Interfaces , 2014, PloS one.

[40]  O Bertrand,et al.  A robust sensor-selection method for P300 brain–computer interfaces , 2011, Journal of neural engineering.

[41]  Tobias Kaufmann,et al.  Comparison of tactile, auditory, and visual modality for brain-computer interface use: a case study with a patient in the locked-in state , 2013, Front. Neurosci..

[42]  Jun Ueda,et al.  Wearable Sensorimotor Enhancer for Fingertip Based on Stochastic Resonance Effect , 2013, IEEE Transactions on Human-Machine Systems.

[43]  Derek K. Jones,et al.  Perisylvian language networks of the human brain , 2005, Annals of neurology.

[44]  J. Buford,et al.  Wavelet methodology to improve single unit isolation in primary motor cortex cells , 2015, Journal of Neuroscience Methods.

[45]  F. Cincotti,et al.  Eye-gaze independent EEG-based brain–computer interfaces for communication , 2012, Journal of neural engineering.

[46]  A. Kübler,et al.  Toward brain-computer interface based wheelchair control utilizing tactually-evoked event-related potentials , 2014, Journal of NeuroEngineering and Rehabilitation.

[47]  Marc M. Van Hulle,et al.  Comparison of Classification Methods for P300 Brain-Computer Interface on Disabled Subjects , 2011, Comput. Intell. Neurosci..

[48]  Xingyu Wang,et al.  An ERP-Based BCI using an oddball Paradigm with Different Faces and Reduced errors in Critical Functions , 2014, Int. J. Neural Syst..

[49]  Isao Nambu,et al.  Estimating the Intended Sound Direction of the User: Toward an Auditory Brain-Computer Interface Using Out-of-Head Sound Localization , 2013, PloS one.

[50]  Shangkai Gao,et al.  An Auditory Brain–Computer Interface Using Active Mental Response , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[51]  Bo Hong,et al.  Employing an active mental task to enhance the performance of auditory attention-based brain–computer interfaces , 2013, Clinical Neurophysiology.

[52]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[53]  N. Birbaumer,et al.  BCI2000: a general-purpose brain-computer interface (BCI) system , 2004, IEEE Transactions on Biomedical Engineering.

[54]  M. Hallett,et al.  Asymmetric spatiotemporal patterns of event-related desynchronization preceding voluntary sequential finger movements: a high-resolution EEG study , 2005, Clinical Neurophysiology.

[55]  A Z Snyder,et al.  Steady-state vibration evoked potentials: descriptions of technique and characterization of responses. , 1992, Electroencephalography and clinical neurophysiology.

[56]  J J Vidal,et al.  Toward direct brain-computer communication. , 1973, Annual review of biophysics and bioengineering.

[57]  D. Hu,et al.  Gaze independent brain–computer speller with covert visual search tasks , 2011, Clinical Neurophysiology.

[58]  F Babiloni,et al.  A comparison of classification techniques for a gaze-independent P300-based brain-computer interface. , 2012, Journal of neural engineering.

[59]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[60]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[61]  Ben H. Jansen,et al.  An exploratory study of factors affecting single trial P300 detection , 2004, IEEE Transactions on Biomedical Engineering.

[62]  Rajesh P. N. Rao,et al.  Distributed cortical adaptation during learning of a brain–computer interface task , 2013, Proceedings of the National Academy of Sciences.

[63]  Gabriel Curio,et al.  Brain-computer communication and slow cortical potentials , 2004, IEEE Transactions on Biomedical Engineering.

[64]  Marcel Kinsbourne,et al.  Attention and the right-ear advantage: What is the connection? , 2011, Brain and Cognition.

[65]  Kiralee M. Hayashi,et al.  3D mapping of language networks in clinical and pre-clinical Alzheimer’s disease , 2008, Brain and Language.

[66]  J. Buford,et al.  Brain–Computer Interface after Nervous System Injury , 2014, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[67]  Touradj Ebrahimi,et al.  An efficient P300-based brain–computer interface for disabled subjects , 2008, Journal of Neuroscience Methods.

[68]  M. Castelo‐Branco,et al.  Comparison of a row-column speller vs. a novel lateral single-character speller: Assessment of BCI for severe motor disabled patients , 2012, Clinical Neurophysiology.

[69]  B. Blankertz,et al.  A New Auditory Multi-Class Brain-Computer Interface Paradigm: Spatial Hearing as an Informative Cue , 2010, PloS one.

[70]  Frédéric E Theunissen,et al.  What's that sound? Auditory area CLM encodes stimulus surprise, not intensity or intensity changes. , 2008, Journal of neurophysiology.

[71]  J. Buford,et al.  Combined corticospinal and reticulospinal effects on upper limb muscles , 2014, Neuroscience Letters.

[72]  Weidong Zhou,et al.  Epileptic Seizure Detection Using Lacunarity and Bayesian Linear Discriminant Analysis in Intracranial EEG , 2013, IEEE Transactions on Biomedical Engineering.

[73]  Benjamin Schrauwen,et al.  A Unified Probabilistic Approach to Improve Spelling in an Event-Related Potential-Based Brain–Computer Interface , 2013, IEEE Transactions on Biomedical Engineering.

[74]  Xiaorong Gao,et al.  An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method , 2009, Journal of neural engineering.

[75]  W. A. Sarnacki,et al.  Electroencephalographic (EEG) control of three-dimensional movement , 2010, Journal of neural engineering.

[76]  Brendan Z. Allison,et al.  P300 brain computer interface: current challenges and emerging trends , 2012, Front. Neuroeng..

[77]  J. Niemi,et al.  Prolonged Mechanical Noise Restores Tactile Sense in Diabetic Neuropathic Patients , 2009, The international journal of lower extremity wounds.