Language Model Applications to Spelling with Brain-Computer Interfaces

Within the Ambient Assisted Living (AAL) community, Brain-Computer Interfaces (BCIs) have raised great hopes as they provide alternative communication means for persons with disabilities bypassing the need for speech and other motor activities. Although significant advancements have been realized in the last decade, applications of language models (e.g., word prediction, completion) have only recently started to appear in BCI systems. The main goal of this article is to review the language model applications that supplement non-invasive BCI-based communication systems by discussing their potential and limitations, and to discern future trends. First, a brief overview of the most prominent BCI spelling systems is given, followed by an in-depth discussion of the language models applied to them. These language models are classified according to their functionality in the context of BCI-based spelling: the static/dynamic nature of the user interface, the use of error correction and predictive spelling, and the potential to improve their classification performance by using language models. To conclude, the review offers an overview of the advantages and challenges when implementing language models in BCI-based communication systems when implemented in conjunction with other AAL technologies.

[1]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces , 2007, Journal of neural engineering.

[2]  Benjamin Blankertz,et al.  THE BERLIN BRAIN-COMPUTER INTERFACE PRESENTS THE NOVEL MENTAL TYPEWRITER HEX-O-SPELL , 2006 .

[3]  Miguel P. Eckstein,et al.  Impact of target probability on single-trial EEG target detection in a difficult rapid serial visual presentation task , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[4]  F. L. D. Silva,et al.  Event-related EEG/MEG synchronization and desynchronization: basic principles , 1999, Clinical Neurophysiology.

[5]  J. W. Minett,et al.  Optimizing the P300-based brain–computer interface: current status, limitations and future directions , 2011, Journal of neural engineering.

[6]  Katrien Vanderperren,et al.  Steady State Visual Evoked Potential (SSVEP) - Based Brain Spelling System with Synchronous and Asynchronous Typing Modes , 2011 .

[7]  S. Gielen,et al.  The brain–computer interface cycle , 2009, Journal of neural engineering.

[8]  Yijun Wang,et al.  A high-speed BCI based on code modulation VEP , 2011, Journal of neural engineering.

[9]  Cuntai Guan,et al.  High performance P300 speller for brain-computer interface , 2004, IEEE International Workshop on Biomedical Circuits and Systems, 2004..

[10]  N. Galloway Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine , 1990 .

[11]  Christoph Guger,et al.  A BCI using VEP for continuous control of a mobile robot , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[12]  Alan F. Blackwell,et al.  Dasher—a data entry interface using continuous gestures and language models , 2000, UIST '00.

[13]  Bo Hong,et al.  A practical VEP-based brain-computer interface , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[14]  Nikola B. Serbedzija Reflective assistance for eldercare environments , 2010, SEHC '10.

[15]  S. Coyle,et al.  Brain–computer interfaces: a review , 2003 .

[16]  Brendan Z. Allison,et al.  Brain-Computer Interfaces: Revolutionizing Human-Computer Interaction , 2013 .

[17]  Arne Robben,et al.  Steady-State Visual Evoked Potential-Based Computer Gaming on a Consumer-Grade EEG Device , 2013, IEEE Transactions on Computational Intelligence and AI in Games.

[18]  Ivan Volosyak,et al.  Evaluation of the Bremen SSVEP based BCI in real world conditions , 2009, 2009 IEEE International Conference on Rehabilitation Robotics.

[19]  José del R. Millán,et al.  General Signal Processing and Machine Learning Tools for BCI Analysis , 2007 .

[20]  A. Schwartz,et al.  High-performance neuroprosthetic control by an individual with tetraplegia , 2013, The Lancet.

[21]  E Donchin,et al.  Brain-computer interface technology: a review of the first international meeting. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[22]  D. Regan Human brain electrophysiology: Evoked potentials and evoked magnetic fields in science and medicine , 1989 .

[23]  N. Thakor,et al.  Journal of Neuroengineering and Rehabilitation Open Access a Brain-computer Interface with Vibrotactile Biofeedback for Haptic Information , 2007 .

[24]  Jan B. F. van Erp,et al.  A Tactile P300 Brain-Computer Interface , 2010, Front. Neurosci..

[25]  J. Wolpaw,et al.  A novel P300-based brain–computer interface stimulus presentation paradigm: Moving beyond rows and columns , 2010, Clinical Neurophysiology.

[26]  Yijun Wang,et al.  VEP-based brain-computer interfaces: time, frequency, and code modulations [Research Frontier] , 2009, IEEE Computational Intelligence Magazine.

[27]  Nader Pouratian,et al.  Natural language processing with dynamic classification improves P300 speller accuracy and bit rate , 2012, Journal of neural engineering.

[28]  Stefano Federici,et al.  Toward functioning and usable brain–computer interfaces (BCIs): A literature review , 2012, Disability and rehabilitation. Assistive technology.

[29]  Ivan Volosyak,et al.  A Dictionary-Driven SSVEP Speller with a Modified Graphical User Interface , 2011, IWANN.

[30]  Arne Robben,et al.  Sampled sinusoidal stimulation profile and multichannel fuzzy logic classification for monitor-based phase-coded SSVEP brain–computer interfacing , 2013, Journal of neural engineering.

[31]  Wolfgang Rosenstiel,et al.  Online Adaptation of a c-VEP Brain-Computer Interface(BCI) Based on Error-Related Potentials and Unsupervised Learning , 2012, PloS one.

[32]  Xingyu Wang,et al.  A combined brain–computer interface based on P300 potentials and motion-onset visual evoked potentials , 2012, Journal of Neuroscience Methods.

[33]  Rabab K Ward,et al.  A survey of signal processing algorithms in brain–computer interfaces based on electrical brain signals , 2007, Journal of neural engineering.

[34]  R.G. Radwin,et al.  Neural Signal Based Control of the Dasher Writing System , 2007, 2007 3rd International IEEE/EMBS Conference on Neural Engineering.

[35]  Hongzhi Qi,et al.  EEG feature comparison and classification of simple and compound limb motor imagery , 2013, Journal of NeuroEngineering and Rehabilitation.

[36]  Günter Edlinger,et al.  Brain-Computer Interface: Generic Control Interface for Social Interaction Applications , 2011, IWANN.

[37]  Luca Citi,et al.  Defining brain–machine interface applications by matching interface performance with device requirements , 2008, Journal of Neuroscience Methods.

[38]  Jonathan R Wolpaw,et al.  A brain-computer interface for long-term independent home use , 2010, Amyotrophic lateral sclerosis : official publication of the World Federation of Neurology Research Group on Motor Neuron Diseases.

[39]  Arne Robben,et al.  Decoding SSVEP Responses using Time Domain Classification , 2018, IJCCI.

[40]  F. Babiloni,et al.  GeoSpell: an alternative P300-based speller interface towards no eye gaze required , 2011 .

[41]  Klaus-Robert Müller,et al.  The non-invasive Berlin Brain–Computer Interface: Fast acquisition of effective performance in untrained subjects , 2007, NeuroImage.

[42]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[43]  Brendan Z. Allison,et al.  Brain-Computer Interfaces: A Gentle Introduction , 2009 .

[44]  H. S. Venkatagiri,et al.  Effect of window size on rate of communication in a lexical prediction AAC system , 1994 .

[45]  Moritz Grosse-Wentrup,et al.  Critical issues in state-of-the-art brain–computer interface signal processing , 2011, Journal of neural engineering.

[46]  G Pfurtscheller,et al.  Graphical display and statistical evaluation of event-related desynchronization (ERD). , 1977, Electroencephalography and clinical neurophysiology.

[47]  Kwee-Bo Sim,et al.  Analysis the effect of PCA for feature reduction in non-stationary EEG based motor imagery of BCI system , 2014 .

[48]  W. Marslen-Wilson Functional parallelism in spoken word-recognition , 1987, Cognition.

[49]  Tobias Kaufmann,et al.  Spelling is Just a Click Away – A User-Centered Brain–Computer Interface Including Auto-Calibration and Predictive Text Entry , 2012, Front. Neurosci..

[50]  Tomohiro Yoshikawa,et al.  Improvement of spelling speed in P300 speller using transition probability of letters , 2012, The 6th International Conference on Soft Computing and Intelligent Systems, and The 13th International Symposium on Advanced Intelligence Systems.

[51]  Benjamin Blankertz,et al.  Exploring motion VEPs for gaze-independent communication , 2012, Journal of neural engineering.

[52]  Brendan Z. Allison,et al.  Context-Awareness as an Enhancement of Brain-Computer Interfaces , 2011, IWAAL.

[53]  J. Wolpaw,et al.  Clinical Applications of Brain-Computer Interfaces: Current State and Future Prospects , 2009, IEEE Reviews in Biomedical Engineering.

[54]  Benjamin Blankertz,et al.  A Novel 9-Class Auditory ERP Paradigm Driving a Predictive Text Entry System , 2011, Front. Neurosci..

[55]  Gary Garcia-Molina,et al.  Optimal spatial filtering for the steady state visual evoked potential: BCI application , 2011, 2011 5th International IEEE/EMBS Conference on Neural Engineering.

[56]  H. Cecotti,et al.  Spelling with Brain-Computer Interfaces Current trends and prospects , 2010 .

[57]  Johan A. K. Suykens,et al.  P300 Detection Based on Feature Extraction in On-line Brain-Computer Interface , 2009, KI.

[58]  Brian Roark,et al.  Fusion with language models improves spelling accuracy for ERP-based brain computer interface spellers , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[59]  Ivan Volosyak,et al.  Evaluation of an SSVEP based Brain-Computer Interface on the command and application levels , 2009, 2009 4th International IEEE/EMBS Conference on Neural Engineering.

[60]  Michel Verleysen,et al.  Feature Selection for Interpatient Supervised Heart Beat Classification , 2011, BIOSIGNALS.

[61]  S. Heinrich A primer on motion visual evoked potentials , 2007, Documenta Ophthalmologica.

[62]  J J Vidal,et al.  Toward direct brain-computer communication. , 1973, Annual review of biophysics and bioengineering.

[63]  Wolfgang Grodd,et al.  Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI) , 2004, IEEE Transactions on Biomedical Engineering.

[64]  M. Kuba,et al.  Motion-onset VEPs: Characteristics, methods, and diagnostic use , 2007, Vision Research.

[65]  Rupert Ortner,et al.  SSVEP Based Brain-Computer Interface for Robot Control , 2010, ICCHP.

[66]  Sercan Taha Ahi,et al.  A Dictionary-Driven P300 Speller With a Modified Interface , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[67]  Dezhong Yao,et al.  L1 Norm based common spatial patterns decomposition for scalp EEG BCI , 2013, BioMedical Engineering OnLine.

[68]  Benjamin Schrauwen,et al.  A P300 BCI for the Masses: Prior Information Enables Instant Unsupervised Spelling , 2012, NIPS.

[69]  J R Wolpaw,et al.  Spatial filter selection for EEG-based communication. , 1997, Electroencephalography and clinical neurophysiology.

[70]  Tae-Seong Kim,et al.  A novel P300-based BCI system for words typing , 2013, 2013 International Winter Workshop on Brain-Computer Interface (BCI).

[71]  Henry Beker,et al.  Cipher Systems: The Protection of Communications , 1982 .

[72]  S.A. Wills,et al.  DASHER-an efficient writing system for brain-computer interfaces? , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[73]  Reza Fazel-Rezai,et al.  A region-based P300 speller for brain-computer interface , 2009, Canadian Journal of Electrical and Computer Engineering.

[74]  J. Schweitzer,et al.  Owner reports of attention, activity, and impulsivity in dogs: a replication study , 2010, Behavioral and Brain Functions.

[75]  Toshihisa Tanaka,et al.  Simultaneous Design of FIR Filter Banks and Spatial Patterns for EEG Signal Classification , 2013, IEEE Transactions on Biomedical Engineering.

[76]  Marc M. Van Hulle,et al.  Comparison of Classification Methods for P300 Brain-Computer Interface on Disabled Subjects , 2011, Comput. Intell. Neurosci..

[77]  M. Matteucci,et al.  The Utility Metric: A Novel Method to Assess the Overall Performance of Discrete Brain–Computer Interfaces , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[78]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern (FBCSP) algorithm using online adaptive and semi-supervised learning , 2011, The 2011 International Joint Conference on Neural Networks.

[79]  G. Pfurtscheller Event-related synchronization (ERS): an electrophysiological correlate of cortical areas at rest. , 1992, Electroencephalography and clinical neurophysiology.

[80]  O. Yanez-Suarez,et al.  SSVEP-based BCI control of the DASHER writing system , 2011, 2011 5th International IEEE/EMBS Conference on Neural Engineering.

[81]  Ivan Volosyak,et al.  SSVEP-based Bremen–BCI interface—boosting information transfer rates , 2011, Journal of neural engineering.

[82]  Shirley M Coyle,et al.  Brain–computer interface using a simplified functional near-infrared spectroscopy system , 2007, Journal of neural engineering.

[83]  Ivan Volosyak,et al.  Impact of Frequency Selection on LCD Screens for SSVEP Based Brain-Computer Interfaces , 2009, IWANN.

[84]  B.Z. Allison,et al.  ERPs evoked by different matrix sizes: implications for a brain computer interface (BCI) system , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[85]  Xingyu Wang,et al.  Spatial-Temporal Discriminant Analysis for ERP-Based Brain-Computer Interface , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[86]  H. S. Wolff,et al.  iRun: Horizontal and Vertical Shape of a Region-Based Graph Compression , 2022, Sensors.

[87]  H H Koester,et al.  Learning and performance of able-bodied individuals using scanning systems with and without word prediction. , 1994, Assistive technology : the official journal of RESNA.

[88]  B. Blankertz,et al.  (C)overt attention and visual speller design in an ERP-based brain-computer interface , 2010, Behavioral and Brain Functions.

[89]  Brendan Z. Allison,et al.  P300 brain computer interface: current challenges and emerging trends , 2012, Front. Neuroeng..

[90]  Marc M. Van Hulle,et al.  Multichannel Decoding for Phase-Coded SSVEP Brain-Computer Interface , 2012, Int. J. Neural Syst..

[91]  Benjamin Blankertz,et al.  Gaze-independent BCI-spelling using rapid serial visual presentation (RSVP) , 2013, Clinical Neurophysiology.

[92]  Arne Robben,et al.  Towards the detection of error-related potentials and its integration in the context of a P300 speller brain-computer interface , 2012, Neurocomputing.

[93]  E Donchin,et al.  The mental prosthesis: assessing the speed of a P300-based brain-computer interface. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[94]  A. Cichocki,et al.  Steady-state visually evoked potentials: Focus on essential paradigms and future perspectives , 2010, Progress in Neurobiology.

[95]  Marcin Kolodziej,et al.  BRAIN-COMPUTER INTERFACE AS MEASUREMENT AND CONTROL SYSTEM THE REVIEW PAPER , 2012 .

[96]  Wolfgang Rosenstiel,et al.  One Class SVM and Canonical Correlation Analysis increase performance in a c-VEP based Brain-Computer Interface (BCI) , 2012, ESANN.

[97]  M. J. E. Salami,et al.  EEG signal classification for real-time brain-computer interface applications: A review , 2011, 2011 4th International Conference on Mechatronics (ICOM).

[98]  Tao Liu,et al.  N200-speller using motion-onset visual response , 2009, Clinical Neurophysiology.

[99]  Jonathan R Wolpaw,et al.  Brain–computer interface systems: progress and prospects , 2007, Expert review of medical devices.

[100]  Ivan Volosyak,et al.  Multiple Channel Detection of Steady-State Visual Evoked Potentials for Brain-Computer Interfaces , 2007, IEEE Transactions on Biomedical Engineering.

[101]  Nikolay Chumerin,et al.  An application of feature selection to on-line P300 detection in brain-computer interface , 2009, 2009 IEEE International Workshop on Machine Learning for Signal Processing.

[102]  B. Allison,et al.  The effects of self-movement, observation, and imagination on mu rhythms and readiness potentials (RP's): toward a brain-computer interface (BCI). , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[103]  L. Cohen,et al.  Brain–computer interfaces: communication and restoration of movement in paralysis , 2007, The Journal of physiology.

[104]  Benjamin Blankertz,et al.  A novel brain-computer interface based on the rapid serial visual presentation paradigm , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[105]  Xiaorong Gao,et al.  A brain–computer interface using motion-onset visual evoked potential , 2008, Journal of neural engineering.

[106]  Xingyu Wang,et al.  P300 Chinese input system based on Bayesian LDA , 2010, Biomedizinische Technik. Biomedical engineering.

[107]  Peter Konrad,et al.  Implantable brain computer interface: Challenges to neurotechnology translation , 2010, Neurobiology of Disease.

[108]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[109]  Mark D. Dunlop,et al.  Predictive text entry methods for mobile phones , 2000, Personal Technologies.

[110]  Eric W. Sellers,et al.  Predictive Spelling With a P300-Based Brain–Computer Interface: Increasing the Rate of Communication , 2010, Int. J. Hum. Comput. Interact..

[111]  Byoung-Kyong Min,et al.  Neuroimaging-based approaches in the brain-computer interface. , 2010, Trends in biotechnology.

[112]  Jonathan R Wolpaw,et al.  Brain–computer interfaces as new brain output pathways , 2007, The Journal of physiology.

[113]  Hubert Cecotti,et al.  Spelling with non-invasive Brain–Computer Interfaces – Current and future trends , 2011, Journal of Physiology-Paris.

[114]  Guillaume Gibert,et al.  xDAWN Algorithm to Enhance Evoked Potentials: Application to Brain–Computer Interface , 2009, IEEE Transactions on Biomedical Engineering.

[115]  Gert Pfurtscheller,et al.  Motor imagery and direct brain-computer communication , 2001, Proc. IEEE.

[116]  Shangkai Gao,et al.  An online brain–computer interface using non-flashing visual evoked potentials , 2010, Journal of neural engineering.

[117]  Xiaorong Gao,et al.  Frequency and Phase Mixed Coding in SSVEP-Based Brain--Computer Interface , 2011, IEEE Transactions on Biomedical Engineering.

[118]  Matteo Matteucci,et al.  A predictive speller controlled by a brain-computer interface based on motor imagery , 2012, TCHI.

[119]  G. Pfurtscheller,et al.  Optimal spatial filtering of single trial EEG during imagined hand movement. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[120]  Yan Wang,et al.  Visual stimulus design for high-rate SSVEP BCI , 2010 .

[121]  Marina Schmid,et al.  An Introduction To The Event Related Potential Technique , 2016 .

[122]  Jonathan Becedas,et al.  Brain–Machine Interfaces: Basis and Advances , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[123]  F Cincotti,et al.  Current trends in hardware and software for brain–computer interfaces (BCIs) , 2011, Journal of neural engineering.

[124]  Howard Bowman,et al.  The cost of space independence in P300-BCI spellers , 2013, Journal of NeuroEngineering and Rehabilitation.