How are visual words represented? Insights from EEG‐based visual word decoding, feature derivation and image reconstruction

Investigations into the neural basis of reading have shed light on the cortical locus and the functional role of visual‐orthographic processing. Yet, the fine‐grained structure of neural representations subserving reading remains to be clarified. Here, we capitalize on the spatiotemporal structure of electroencephalography (EEG) data to examine if and how EEG patterns can serve to decode and reconstruct the internal representation of visually presented words in healthy adults. Our results show that word classification and image reconstruction were accurate well above chance, that their temporal profile exhibited an early onset, soon after 100 ms, and peaked around 170 ms. Further, reconstruction results were well explained by a combination of visual‐orthographic word properties. Last, systematic individual differences were detected in orthographic representations across participants. Collectively, our results establish the feasibility of EEG‐based word decoding and image reconstruction. More generally, they help to elucidate the specific features, dynamics, and neurocomputational principles underlying word recognition.

[1]  Amir Amedi,et al.  Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind , 2012, Neuron.

[2]  A Mouraux,et al.  Across-trial averaging of event-related EEG responses and beyond. , 2008, Magnetic resonance imaging.

[3]  Sadato Norihiro,et al.  Visual image reconstruction from human brain activity , 2009 .

[4]  A. Meyers Reading , 1999, Language Teaching.

[5]  Grzegorz Kondrak,et al.  On the Syllabification of Phonemes , 2009, NAACL.

[6]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[7]  R. Smits,et al.  Patterns of English phoneme confusions by native and non-native listeners. , 2004, The Journal of the Acoustical Society of America.

[8]  Colin M. Brown,et al.  Anticipating upcoming words in discourse: evidence from ERPs and reading times. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[9]  S. Dehaene,et al.  The unique role of the visual word form area in reading , 2011, Trends in Cognitive Sciences.

[10]  Charles A. Perfetti,et al.  Reading Ability: Lexical Quality to Comprehension , 2007 .

[11]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[12]  D. Share On the Anglocentricities of current reading research and practice: the perils of overreliance on an "outlier" orthography. , 2008, Psychological bulletin.

[13]  Laurie S. Glezer,et al.  Evidence for Highly Selective Neuronal Tuning to Whole Words in the “Visual Word Form Area” , 2009, Neuron.

[14]  Jonathan Grainger,et al.  A Thousand Words Are Worth a Picture , 2015, Psychological science.

[15]  Kara D. Federmeier,et al.  The N400 as a snapshot of interactive processing: Evidence from regression analyses of orthographic neighbor and lexical associate effects. , 2011, Psychophysiology.

[16]  Richard F Murray,et al.  Classification images: A review. , 2011, Journal of vision.

[17]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[18]  Doris Y. Tsao,et al.  The Code for Facial Identity in the Primate Brain , 2017, Cell.

[19]  Brice A. Kuhl,et al.  Neural portraits of perception: Reconstructing face images from evoked brain activity , 2014, NeuroImage.

[20]  D G Pelli,et al.  The VideoToolbox software for visual psychophysics: transforming numbers into movies. , 1997, Spatial vision.

[21]  Susan G. Wardle,et al.  Decoding Dynamic Brain Patterns from Evoked Responses: A Tutorial on Multivariate Pattern Analysis Applied to Time Series Neuroimaging Data , 2016, Journal of Cognitive Neuroscience.

[22]  C. Price,et al.  The Interactive Account of ventral occipitotemporal contributions to reading , 2011, Trends in Cognitive Sciences.

[23]  Matthias Niemeier,et al.  The Neural Dynamics of Facial Identity Processing: Insights from EEG-Based Pattern Analysis and Image Reconstruction , 2018, eNeuro.

[24]  Matthew H. Davis,et al.  Can cognitive models explain brain activation during word and pseudoword reading? A meta-analysis of 36 neuroimaging studies. , 2013, Psychological bulletin.

[25]  A. Nobre,et al.  The anatomy and time course of semantic priming investigated by fMRI and ERPs , 2003, Neuropsychologia.

[26]  R. Salmelin,et al.  Dissociation of normal feature analysis and deficient processing of letter-strings in dyslexic adults. , 1999, Cerebral cortex.

[27]  Guohua Shen,et al.  Deep image reconstruction from human brain activity , 2017, bioRxiv.

[28]  C. Fowler,et al.  Explaining individual differences in reading : theory and evidence , 2011 .

[29]  P Suppes,et al.  Brain wave recognition of words. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[30]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[31]  Karl Magnus Petersson,et al.  Lexical and sublexical orthographic processing: An ERP study with skilled and dyslexic adult readers , 2015, Brain and Language.

[32]  D. Plaut,et al.  The neural basis of visual word form processing: a multivariate investigation. , 2013, Cerebral cortex.

[33]  Daniel Brandeis,et al.  Evidence for developmental changes in the visual word processing network beyond adolescence , 2006, NeuroImage.

[34]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[35]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[36]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[37]  J. Pernier,et al.  ERP Manifestations of Processing Printed Words at Different Psycholinguistic Levels: Time Course and Scalp Distribution , 1999, Journal of Cognitive Neuroscience.

[38]  Manuel Perea,et al.  The processing of consonants and vowels during letter identity and letter position assignment in visual-word recognition: An ERP study , 2011, Brain and Language.

[39]  Friedemann Pulvermüller,et al.  Early Visual Word Processing Is Flexible: Evidence from Spatiotemporal Brain Dynamics , 2015, Journal of Cognitive Neuroscience.

[40]  Tom Heskes,et al.  Linear reconstruction of perceived images from human brain activity , 2013, NeuroImage.

[41]  Jean-Baptiste Poline,et al.  Inverse retinotopy: Inferring the visual content of images from brain activation patterns , 2006, NeuroImage.

[42]  L. Siegel,et al.  Cognitive and linguistic factors in reading acquisition , 2010, Reading and writing.

[43]  Julie A Fiez,et al.  Decoding and disrupting left midfusiform gyrus activity during word reading , 2016, Proceedings of the National Academy of Sciences.

[44]  Brian A. Wandell,et al.  Position sensitivity in the visual word form area , 2012, Proceedings of the National Academy of Sciences.

[45]  Brian N. Pasley,et al.  Reconstructing Speech from Human Auditory Cortex , 2012, PLoS biology.

[46]  J. Zevin,et al.  The impact of task demand on visual word recognition , 2014, Neuroscience.

[47]  M. Posner,et al.  Establishing a time‐line of word recognition: evidence from eye movements and event‐related potentials , 1998, Neuroreport.

[48]  S. Valdois,et al.  New Insights on Developmental Dyslexia Subtypes: Heterogeneity of Mixed Reading Profiles , 2014, PloS one.

[49]  Chris I. Baker,et al.  Influence of lexical status and orthographic similarity on the multi-voxel response of the visual word form area , 2015, NeuroImage.

[50]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[51]  Daniel Brandeis,et al.  Impaired tuning of a fast occipito-temporal response for print in dyslexic children learning to read. , 2007, Brain : a journal of neurology.

[52]  P. Schyns,et al.  Superstitious Perceptions Reveal Properties of Internal Representations , 2003, Psychological science.

[53]  F. Fazio,et al.  Dyslexia: Cultural Diversity and Biological Unity , 2001, Science.

[54]  Marc Brysbaert,et al.  Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English , 2009, Behavior research methods.

[55]  Marlene Behrmann,et al.  Feature-based face representations and image reconstruction from behavioral and neural data , 2015, Proceedings of the National Academy of Sciences.

[56]  Urs Maurer,et al.  Left-lateralized N170 Effects of Visual Expertise in Reading: Evidence from Japanese Syllabic and Logographic Scripts , 2008, Journal of Cognitive Neuroscience.

[57]  Maryellen C. MacDonald,et al.  The Impact of Language Experience on Language and Reading: A Statistical Learning Approach , 2018 .

[58]  Blair C. Armstrong,et al.  The what, when, where, and how of visual word recognition , 2014, Trends in Cognitive Sciences.

[59]  M. Rugg The effects of semantic priming and work repetition on event-related potentials. , 1985, Psychophysiology.

[60]  Sydney S. Cash,et al.  Decoding word and category-specific spatiotemporal representations from MEG and EEG , 2011, NeuroImage.

[61]  Manuel Carreiras,et al.  Universal brain signature of proficient reading: Evidence from four contrasting languages , 2015, Proceedings of the National Academy of Sciences.

[62]  William D. Marslen-Wilson,et al.  The time course of visual word recognition as revealed by linear regression analysis of ERP data , 2006, NeuroImage.