The N400 for brain computer interfacing: complexities and opportunities

The N400 is an Event Related Potential that is evoked in response to conceptually meaningful stimuli. It is for instance more negative in response to incongruent than congruent words in a sentence, and more negative for unrelated than related words following a prime word. This sensitivity to semantic content of a stimulus in relation to the mental context of an individual makes it a signal of interest for Brain Computer Interfaces. A complicating aspect is the number of factors that can affect the N400 amplitude. In this paper, we provide an accessible overview of this range of N400 effects, and survey the three main BCI application areas that currently exploit the N400: (1) exploiting the semantic processing of faces to enhance matrix speller performance, (2) detecting language processing in patients with Disorders of Consciousness, and (3) using semantic stimuli to probe what is on a user's mind. Drawing on studies from these application areas, we illustrate that the N400 can successfully be exploited for BCI purposes, but that the signal-to-noise ratio is a limiting factor, with signal strength also varying strongly across subjects. Furthermore, we put findings in context of the general N400 literature, noting open questions and identifying opportunities for further research.

[1]  Burr Settles,et al.  Active Learning , 2012, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[2]  J. Giacino,et al.  The minimally conscious state: Definition and diagnostic criteria , 2002, Neurology.

[3]  Colin M. Brown,et al.  The N400 as a function of the level of processing. , 1995, Psychophysiology.

[4]  N. Birbaumer,et al.  Brain–computer interfaces and communication in paralysis: Extinction of goal directed thinking in completely paralysed patients? , 2008, Clinical Neurophysiology.

[5]  J. Grainger,et al.  An Electrophysiological Study of the Effects of Orthographic Neighborhood Size on Printed Word Perception , 2002, Journal of Cognitive Neuroscience.

[6]  H. Neville,et al.  Language and , 2019, Adventure Diffusion.

[7]  N. Birbaumer,et al.  Information processing in severe disorders of consciousness: Vegetative state and minimally conscious state , 2005, Clinical Neurophysiology.

[8]  M. Rugg,et al.  Event-related potentials and the semantic matching of pictures , 1990, Brain and Cognition.

[9]  Ellen F. Lau,et al.  A cortical network for semantics: (de)constructing the N400 , 2008, Nature Reviews Neuroscience.

[10]  Norma Wioland,et al.  Cortical Information Processing in Coma , 2009, Cognitive and behavioral neurology : official journal of the Society for Behavioral and Cognitive Neurology.

[11]  Peter,et al.  Semantic Unification , 2008 .

[12]  Mark S. Seidenberg,et al.  Semantic feature production norms for a large set of living and nonliving things , 2005, Behavior research methods.

[13]  Steven Laureys,et al.  Enhancing clinical communication assessments using an audiovisual BCI for patients with disorders of consciousness , 2017, Journal of neural engineering.

[14]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[15]  Shlomo Bentin,et al.  The effects of immediate stimulus repetition on reaction time and event-related potentials in tasks of different complexity , 1994 .

[16]  Jean-Rémi King,et al.  Probing ERP correlates of verbal semantic processing in patients with impaired consciousness , 2015, Neuropsychologia.

[17]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[18]  M. Schlesewsky,et al.  The P600-as-P3 hypothesis revisited: Single-trial analyses reveal that the late EEG positivity following linguistically deviant material is reaction time aligned , 2014, Brain and Language.

[19]  A. Kübler,et al.  Brain–computer interfaces in the continuum of consciousness , 2007, Current opinion in neurology.

[20]  B Blankertz,et al.  Real-time inference of word relevance from electroencephalogram and eye gaze , 2017, Journal of neural engineering.

[21]  Srivas Chennu,et al.  Bedside detection of awareness in the vegetative state: a cohort study , 2011, The Lancet.

[22]  Brendan Z. Allison,et al.  Could Anyone Use a BCI? , 2010, Brain-Computer Interfaces.

[23]  A. Kübler,et al.  Face stimuli effectively prevent brain–computer interface inefficiency in patients with neurodegenerative disease , 2013, Clinical Neurophysiology.

[24]  Vladimir López,et al.  ERPs studies of cognitive processing during sleep. , 2009, International journal of psychology : Journal international de psychologie.

[25]  Christian Mühl,et al.  Guessing what ' s on your mind : Using the N 400 in Brain Computer Interfaces , 2010 .

[26]  Colin M. Brown,et al.  Anticipating upcoming words in discourse: evidence from ERPs and reading times. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[27]  Graeme Hirst,et al.  Evaluating WordNet-based Measures of Lexical Semantic Relatedness , 2006, CL.

[28]  C. Van Petten,et al.  Prediction during language comprehension: benefits, costs, and ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[29]  A. Jacobs,et al.  Frequency and predictability effects on event-related potentials during reading , 2006, Brain Research.

[30]  C. Petten.  A comparison of lexical and sentence-level context effects in event-related potentials , 1993 .

[31]  Rebecca Treiman,et al.  The English Lexicon Project , 2007, Behavior research methods.

[32]  Kara D. Federmeier,et al.  Contextual constraints on lexico-semantic processing in aging: Evidence from single-word event-related brain potentials , 2018, Brain Research.

[33]  Kara D. Federmeier,et al.  The P3b and P600(s): Positive contributions to language comprehension. , 2019, Psychophysiology.

[34]  Thomas A. Schreiber,et al.  The University of South Florida free association, rhyme, and word fragment norms , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[35]  L. Osterhout,et al.  Neural correlates of second-language word learning: minimal instruction produces rapid change , 2004, Nature Neuroscience.

[36]  Walter G Sannita,et al.  Unresponsive wakefulness syndrome: a new name for the vegetative state or apallic syndrome , 2010, BMC medicine.

[37]  Peter Hagoort,et al.  Individual Variation in the Late Positive Complex to Semantic Anomalies , 2012, Front. Psychology.

[38]  Kara D. Federmeier,et al.  The N400 as a snapshot of interactive processing: Evidence from regression analyses of orthographic neighbor and lexical associate effects. , 2011, Psychophysiology.

[39]  L. Deouell,et al.  STRUCTURAL ENCODING AND IDENTIFICATION IN FACE PROCESSING: ERP EVIDENCE FOR SEPARATE MECHANISMS , 2000, Cognitive neuropsychology.

[40]  Ruben G. L. Real,et al.  Towards a more precise neurophysiological assessment of cognitive functions in patients with disorders of consciousness. , 2013, Restorative neurology and neuroscience.

[41]  Brian N. Pasley,et al.  The use of intracranial recordings to decode human language: Challenges and opportunities , 2016, Brain and Language.

[42]  Laurie Zoloth,et al.  Willful modulations of brain activity in disorders of consciousness, by MM Monti, A Vanhaudenhuyse, MR Coleman, M Boly, JD Pickard, L Tshibanda, AM Owen, S Laureys: New England Journal of Medicine , 2010 .

[43]  M. Kutas,et al.  Interactions between sentence context and word frequencyinevent-related brainpotentials , 1990, Memory & cognition.

[44]  Benjamin Blankertz,et al.  Implicit relevance feedback from electroencephalography and eye tracking in image search. , 2018, Journal of neural engineering.

[45]  Peter Desain,et al.  Detecting Semantic Priming at the Single-Trial Level , 2013, PloS one.

[46]  Giosuè Baggio,et al.  The balance between memory and unification in semantics: A dynamic account of the N400 , 2011 .

[47]  Mante S. Nieuwland,et al.  When the Truth Is Not Too Hard to Handle , 2008, Psychological science.

[48]  Gert Storms,et al.  Word associations: Norms for 1,424 Dutch words in a continuous task , 2008, Behavior research methods.

[49]  M. V. Gerven,et al.  Attention modulations of posterior alpha as a control signal for two-dimensional brain–computer interfaces , 2009, Journal of Neuroscience Methods.

[50]  B.Z. Allison,et al.  ERPs evoked by different matrix sizes: implications for a brain computer interface (BCI) system , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[51]  P. Holcomb,et al.  Event-related brain potentials elicited by syntactic anomaly , 1992 .

[52]  Amy Beth Warriner,et al.  Concreteness ratings for 40 thousand generally known English word lemmas , 2014, Behavior research methods.

[53]  M. Rugg,et al.  Event-related potentials and recognition memory for words. , 1989, Electroencephalography and clinical neurophysiology.

[54]  Jeffrey G Malins,et al.  Neuroimage: Clinical the Reliability of the N400 in Single Subjects: Implications for Patients with Disorders of Consciousness , 2022 .

[55]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[56]  M. Boly,et al.  Willful modulation of brain activity in disorders of consciousness. , 2010, The New England journal of medicine.

[57]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[58]  W. Pritchard.  Psychophysiology of P300. , 1981, Psychological bulletin.

[59]  Vladimir Bostanov,et al.  The t-CWT: A new ERP detection and quantification method based on the continuous wavelet transform and Student’s t-statistics , 2006, Clinical Neurophysiology.

[60]  D Deacon,et al.  How automatically is meaning accessed: a review of the effects of attention on semantic processing. , 2000, Frontiers in bioscience : a journal and virtual library.

[61]  Aldenor G. Santos,et al.  Occurrence of the potent mutagens 2- nitrobenzanthrone and 3-nitrobenzanthrone in fine airborne particles , 2019, Scientific Reports.

[62]  Matthew H. Davis,et al.  Detecting awareness in the vegetative state. , 2006, Science.

[63]  Edmund C. Lalor,et al.  Electrophysiological Correlates of Semantic Dissimilarity Reflect the Comprehension of Natural, Narrative Speech , 2017, Current Biology.

[64]  Salim Roukos,et al.  Brain potentials related to stages of sentence verification. , 1983, Psychophysiology.

[65]  Carryl L. Baldwin,et al.  Cloze probability and completion norms for 498 sentences: Behavioral and neural validation using event-related potentials , 2010, Behavior research methods.

[66]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[67]  W. Witzke,et al.  How vegetative is the vegetative state? Preserved semantic processing in VS patients--evidence from N 400 event-related potentials. , 2005, NeuroRehabilitation.

[68]  Marta Kutas,et al.  Harry Potter and the Chamber of What?: the impact of what individuals know on word processing during reading , 2018, Language, cognition and neuroscience.

[69]  James L. McClelland,et al.  Modelling the N400 brain potential as change in a probabilistic representation of meaning , 2018, Nature Human Behaviour.

[70]  Jason Farquhar,et al.  Towards a Communication Brain Computer Interface Based on Semantic Relations , 2014, PloS one.

[71]  L. Aravind,et al.  Integration of Word Meaning and World Knowledge in Language Comprehension , 2022 .

[72]  Heather J. Ferguson,et al.  Dissociable effects of prediction and integration during language comprehension: Evidence from a large-scale study using brain potentials , 2018, bioRxiv.

[73]  C. C. Wood,et al.  Event-related potentials, lexical decision and semantic priming. , 1985, Electroencephalography and clinical neurophysiology.

[74]  L. Otten,et al.  Concreteness in word processing: ERP and behavioral effects in a lexical decision task , 2013, Brain and Language.

[75]  T. Maeda,et al.  Unconscious learning of auditory discrimination using mismatch negativity (MMN) neurofeedback , 2014, Scientific Reports.

[76]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[77]  Jonathan Grainger,et al.  An electrophysiological megastudy of spoken word recognition , 2018, Language, cognition and neuroscience.

[78]  Xingyu Wang,et al.  An ERP-Based BCI using an oddball Paradigm with Different Faces and Reduced errors in Critical Functions , 2014, Int. J. Neural Syst..

[79]  Thomas P. Trappenberg,et al.  A rapid event-related potential (ERP) method for point-of-care evaluation of brain function: Development of the Halifax Consciousness Scanner , 2015, Journal of Neuroscience Methods.

[80]  W. Ritter,et al.  The relationship between N2 and N400: scalp distribution, stimulus probability, and task relevance. , 1991, Psychophysiology.

[81]  Marta Kutas,et al.  Sounds, words, sentences: age-related changes across levels of language processing. , 2003, Psychology and aging.

[82]  M. Rugg.  Event-related brain potentials dissociate repetition effects of high-and low-frequency words , 1990, Memory & cognition.

[83]  M. Rugg,et al.  Event-related potentials and the semantic matching of faces , 1989, Neuropsychologia.

[84]  B. Blankertz,et al.  (C)overt attention and visual speller design in an ERP-based brain-computer interface , 2010, Behavioral and Brain Functions.

[85]  M. Arns,et al.  P300 Development across the Lifespan: A Systematic Review and Meta-Analysis , 2014, PloS one.

[86]  M. Eimer.  Event-related brain potentials distinguish processing stages involved in face perception and recognition , 2000, Clinical Neurophysiology.

[87]  J. Kounios,et al.  Concreteness effects in semantic processing: ERP evidence supporting dual-coding theory. , 1994, Journal of experimental psychology. Learning, memory, and cognition.

[88]  Kara D. Federmeier,et al.  Won't get fooled again: An event-related potential study of task and repetition effects on the semantic processing of items without semantics , 2012, Language and cognitive processes.

[89]  Ali Bahramisharif,et al.  Brain-computer interfacing using modulations of alpha activity induced by covert shifts of attention , 2011, Journal of NeuroEngineering and Rehabilitation.

[90]  A. Kübler,et al.  Flashing characters with famous faces improves ERP-based brain–computer interface performance , 2011, Journal of neural engineering.

[91]  H. Neville,et al.  Visual and auditory sentence processing: A developmental analysis using event‐related brain potentials , 1992 .

[92]  S. Eickhoff,et al.  N400 predicts recovery from disorders of consciousness , 2013, Annals of neurology.