Listen, You are Writing! Speeding up Online Spelling with a Dynamic Auditory BCI

Representing an intuitive spelling interface for brain–computer interfaces (BCI) in the auditory domain is not straight-forward. In consequence, all existing approaches based on event-related potentials (ERP) rely at least partially on a visual representation of the interface. This online study introduces an auditory spelling interface that eliminates the necessity for such a visualization. In up to two sessions, a group of healthy subjects (N = 21) was asked to use a text entry application, utilizing the spatial cues of the AMUSE paradigm (Auditory Multi-class Spatial ERP). The speller relies on the auditory sense both for stimulation and the core feedback. Without prior BCI experience, 76% of the participants were able to write a full sentence during the first session. By exploiting the advantages of a newly introduced dynamic stopping method, a maximum writing speed of 1.41 char/min (7.55 bits/min) could be reached during the second session (average: 0.94 char/min, 5.26 bits/min). For the first time, the presented work shows that an auditory BCI can reach performances similar to state-of-the-art visual BCIs based on covert attention. These results represent an important step toward a purely auditory BCI.

[1]  J. Cedarbaum,et al.  The ALSFRS-R: a revised ALS functional rating scale that incorporates assessments of respiratory function , 1999, Journal of the Neurological Sciences.

[2]  M S Treder,et al.  Gaze-independent brain–computer interfaces based on covert attention and feature attention , 2011, Journal of neural engineering.

[3]  Bernhard Schölkopf,et al.  An Auditory Paradigm for Brain-Computer Interfaces , 2004, NIPS.

[4]  P. Tonin,et al.  P300-Based Brain–Computer Interface Communication: Evaluation and Follow-up in Amyotrophic Lateral Sclerosis , 2009, Front. Neuropro..

[5]  Ko-ichiro Miyamoto,et al.  A brain-computer interface (BCI) system based on auditory stream segregation , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[6]  Cuntai Guan,et al.  Asynchronous P300-Based Brain--Computer Interfaces: A Computational Approach With Statistical Models , 2008, IEEE Transactions on Biomedical Engineering.

[7]  D. M. Green,et al.  Sound localization by human listeners. , 1991, Annual review of psychology.

[8]  Benjamin Blankertz,et al.  THE BERLIN BRAIN-COMPUTER INTERFACE PRESENTS THE NOVEL MENTAL TYPEWRITER HEX-O-SPELL , 2006 .

[9]  Dean J Krusienski,et al.  A comparison of classification techniques for the P300 Speller , 2006, Journal of neural engineering.

[10]  P Desain,et al.  Sequenced subjective accents for brain–computer interfaces , 2011, Journal of neural engineering.

[11]  A Belitski,et al.  P300 audio-visual speller , 2011, Journal of neural engineering.

[12]  B. Schoelkopf,et al.  Transition from the locked in to the completely locked-in state: A physiological analysis , 2011, Clinical Neurophysiology.

[13]  José del R. Millán,et al.  Evaluation Criteria for BCI Research , 2007 .

[14]  E. Donchin,et al.  A P300-based brain–computer interface: Initial tests by ALS patients , 2006, Clinical Neurophysiology.

[15]  H. Flor,et al.  A multimodal brain-based feedback and communication system , 2004, Experimental Brain Research.

[16]  Febo Cincotti,et al.  Vibrotactile Feedback for Brain-Computer Interface Operation , 2007, Comput. Intell. Neurosci..

[17]  J. Wolpaw,et al.  A P300-based brain–computer interface for people with amyotrophic lateral sclerosis , 2008, Clinical Neurophysiology.

[18]  N. Birbaumer,et al.  An auditory oddball brain–computer interface for binary choices , 2010, Clinical Neurophysiology.

[19]  B. Blankertz,et al.  (C)overt attention and visual speller design in an ERP-based brain-computer interface , 2010, Behavioral and Brain Functions.

[20]  J. Polich,et al.  P300 from auditory and somatosensory stimuli: probability and inter-stimulus interval. , 1991, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[21]  J. Wolpaw,et al.  Does the ‘P300’ speller depend on eye gaze? , 2010, Journal of neural engineering.

[22]  Jonathan R Wolpaw,et al.  A brain-computer interface for long-term independent home use , 2010, Amyotrophic lateral sclerosis : official publication of the World Federation of Neurology Research Group on Motor Neuron Diseases.

[23]  C. Neuper,et al.  Toward a high-throughput auditory P300-based brain–computer interface , 2009, Clinical Neurophysiology.

[24]  N. Birbaumer,et al.  An auditory oddball (P300) spelling system for brain-computer interfaces. , 2009, Psychophysiology.

[25]  Brendan Z Allison,et al.  Effects of SOA and flash pattern manipulations on ERPs, performance, and preference: implications for a BCI system. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[26]  B. Blankertz,et al.  Initial results of a high-speed spatial auditory BCI , 2009 .

[27]  I. Magnano,et al.  Visual and auditory event-related potentials in sporadic amyotrophic lateral sclerosis , 2002, Clinical Neurophysiology.

[28]  E. Langendijk,et al.  Contribution of spectral cues to human sound localization. , 1999, The Journal of the Acoustical Society of America.

[29]  B. Blankertz,et al.  Online detection of error potentials increases information throughput in a brain–computer interface , 2011, Neuroscience Letters.

[30]  C. Rennie,et al.  Decrement of the N1 auditory event-related potential with stimulus repetition: habituation vs. refractoriness. , 1998, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[31]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[32]  J. Polich,et al.  P300 amplitude is determined by target-to-target interval. , 2002, Psychophysiology.

[33]  J. Huggins,et al.  What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis , 2011, Amyotrophic lateral sclerosis : official publication of the World Federation of Neurology Research Group on Motor Neuron Diseases.

[34]  J. Blauert Spatial Hearing: The Psychophysics of Human Sound Localization , 1983 .

[35]  Robin J. Caster Handbook of the Psychology of Aging, 5th ed. , 2003 .

[36]  F. Carp Handbook of the psychology of aging. , 1977 .

[37]  A. Lenhardt,et al.  An Adaptive P300-Based Online Brain–Computer Interface , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[38]  Stefano F. Cappa,et al.  Auditory event-related potentials in non-demented patients with sporadic amyotrophic lateral sclerosis , 2008, Clinical Neurophysiology.

[39]  A. Kübler,et al.  A Brain–Computer Interface Controlled Auditory Event‐Related Potential (P300) Spelling System for Locked‐In Patients , 2009, Annals of the New York Academy of Sciences.

[40]  Shangkai Gao,et al.  An Auditory Brain–Computer Interface Using Active Mental Response , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[41]  B. Blankertz,et al.  A New Auditory Multi-Class Brain-Computer Interface Paradigm: Spatial Hearing as an Informative Cue , 2010, PloS one.

[42]  E. Sellers,et al.  How many people are able to control a P300-based brain–computer interface (BCI)? , 2009, Neuroscience Letters.

[43]  Stefan Haufe,et al.  The Berlin Brain–Computer Interface: Non-Medical Uses of BCI Technology , 2010, Front. Neurosci..

[44]  Marc Schröder,et al.  The German Text-to-Speech Synthesis System MARY: A Tool for Research, Development and Teaching , 2003, Int. J. Speech Technol..

[45]  Christa Neuper,et al.  Impact of auditory distraction on user performance in a brain–computer interface driven by different mental tasks , 2011, Clinical Neurophysiology.

[46]  D. McFarland,et al.  An auditory brain–computer interface (BCI) , 2008, Journal of Neuroscience Methods.

[47]  Klaus-Robert Müller,et al.  Playing Pinball with non-invasive BCI , 2008, NIPS.

[48]  Benjamin Blankertz,et al.  A novel brain-computer interface based on the rapid serial visual presentation paradigm , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[49]  G. Pfurtscheller,et al.  EEG-based communication: presence of an error potential , 2000, Clinical Neurophysiology.

[50]  D. Hu,et al.  Gaze independent brain–computer speller with covert visual search tasks , 2011, Clinical Neurophysiology.

[51]  D. Grantham,et al.  Auditory spatial resolution in horizontal, vertical, and diagonal planes. , 2003, The Journal of the Acoustical Society of America.

[52]  A. Oxenham,et al.  Influence of musical and psychoacoustical training on pitch discrimination , 2006, Hearing Research.

[53]  Stefan Haufe,et al.  Single-trial analysis and classification of ERP components — A tutorial , 2011, NeuroImage.

[54]  Anton Nijholt,et al.  Turning Shortcomings into Challenges: Brain-Computer Interfaces for Games , 2009, INTETAIN.

[55]  Chang-Hwan Im,et al.  Classification of selective attention to auditory stimuli: Toward vision-free brain–computer interfacing , 2011, Journal of Neuroscience Methods.

[56]  G.F. Inbar,et al.  An improved P300-based brain-computer interface , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[57]  J. Fozard,et al.  Changes in vision and hearing with aging. , 2001 .

[58]  Simon Carlile,et al.  The nature and distribution of errors in sound localization by human listeners , 1997, Hearing Research.

[59]  T. Demiralp,et al.  Cognitive impairment in amyotrophic lateral sclerosis: evidence from neuropsychological investigation and event-related potentials. , 2002, Brain research. Cognitive brain research.

[60]  Benjamin Blankertz,et al.  A Novel 9-Class Auditory ERP Paradigm Driving a Predictive Text Entry System , 2011, Front. Neurosci..

[61]  C. Gonsalvez,et al.  Target-to-target interval, intensity, and P300 from an auditory single-stimulus task. , 2007, Psychophysiology.

[62]  Jan B. F. van Erp,et al.  A Tactile P300 Brain-Computer Interface , 2010, Front. Neurosci..

[63]  F L Wightman,et al.  Resolution of front-back ambiguity in spatial hearing by listener and source movement. , 1999, The Journal of the Acoustical Society of America.

[64]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[65]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[66]  A. Kübler,et al.  Brain Painting: First Evaluation of a New Brain–Computer Interface Application with ALS-Patients and Healthy Volunteers , 2010, Front. Neurosci..

[67]  Shangkai Gao,et al.  An online brain–computer interface using non-flashing visual evoked potentials , 2010, Journal of neural engineering.

[68]  Benjamin Blankertz,et al.  Two-dimensional auditory p300 speller with predictive text system , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.