No evidence from MVPA for different processes underlying the N300 and N400 incongruity effects in object-scene processing

Attributing meaning to diverse visual input is a core feature of human cognition. Violating environmental expectations (e.g., a toothbrush in the fridge) induces a late event-related negativity of the event-related potential/ERP. This N400 ERP has not only been linked to the semantic processing of language, but also to objects and scenes. Inconsistent object-scene relationships are additionally associated with an earlier negative deflection of the EEG signal between 250 and 350 ms. This N300 is hypothesized to reflect pre-semantic perceptual processes. To investigate whether these two components are truly separable or if the early object-scene integration activity (250-350 ms) shares certain levels of processing with the late neural correlates of meaning processing (350-500 ms), we used time-resolved multivariate pattern analysis (MVPA) where a classifier trained at one time point in a trial (e.g., during the N300 time window) is tested at every other time point (i.e., including the N400 time window). Forty participants were presented with semantic inconsistencies, in which an object was inconsistent with a scene's meaning. Replicating previous findings, our manipulation produced significant N300 and N400 deflections. MVPA revealed above chance decoding performance for classifiers trained during time points of the N300 component and tested during later time points of the N400, and vice versa. This provides no evidence for the activation of two separable neurocognitive processes following the violation of context-dependent predictions in visual scene perception. Our data supports the early appearance of high-level, context-sensitive processes in visual cognition.

[1]  Haline E. Schendan,et al.  Object knowledge during entry-level categorization is activated and modified by implicit memory after 200 ms , 2009, NeuroImage.

[2]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[3]  Tim H. W. Cornelissen,et al.  Stuck on semantics: Processing of irrelevant object-scene inconsistencies modulates ongoing gaze behavior , 2017, Attention, perception & psychophysics.

[4]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[5]  Stephen M. Smith,et al.  Threshold-free cluster enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference , 2009, NeuroImage.

[6]  G. McCarthy,et al.  Language-Related ERPs: Scalp Distributions and Modulation by Word Type and Semantic Priming , 1994, Journal of Cognitive Neuroscience.

[7]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[8]  M. Bar,et al.  Prediction, context, and competition in visual recognition , 2015, Annals of the New York Academy of Sciences.

[9]  M. Kutas,et al.  The Search for Common Sense: An Electrophysiological Study of the Comprehension of Words and Pictures in Reading , 1996, Journal of Cognitive Neuroscience.

[10]  Liad Mudrik,et al.  Are incongruent objects harder to identify? The functional significance of the N300 component , 2018, Neuropsychologia.

[11]  Niko A. Busch,et al.  A Pool of Pairs of Related Objects (POPORO) for Investigating Visual Semantic Integration: Behavioral and Electrophysiological Validation , 2012, Brain Topography.

[12]  I. Biederman,et al.  Scene perception: Detecting and judging objects undergoing relational violations , 1982, Cognitive Psychology.

[13]  M. Castelhano,et al.  Scene context influences without scene gist: Eye movements guided by spatial associations in visual search , 2011, Psychonomic bulletin & review.

[14]  Moshe Bar,et al.  The proactive brain: memory for predictions , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[15]  Tim Lauer,et al.  The role of scene summary statistics in object recognition , 2018, Scientific Reports.

[16]  J. Henderson,et al.  Object identification is isolated from scene semantic constraint: evidence from object type and token discrimination. , 1999, Acta psychologica.

[17]  Jan Theeuwes,et al.  OpenSesame: An open-source, graphical experiment builder for the social sciences , 2011, Behavior Research Methods.

[18]  Martin Luessi,et al.  MEG and EEG data analysis with MNE-Python , 2013, Front. Neuroinform..

[19]  Alexandre Gramfort,et al.  Autoreject: Automated artifact rejection for MEG and EEG data , 2016, NeuroImage.

[20]  N. Mackworth,et al.  Cognitive determinants of fixation location during picture viewing. , 1978, Journal of experimental psychology. Human perception and performance.

[21]  M. Kutas,et al.  Neurophysiological evidence for two processing times for visual object identification , 2002, Neuropsychologia.

[22]  J. Henderson,et al.  The effects of semantic consistency on eye movements during complex scene viewing , 1999 .

[23]  C. Fiebach,et al.  Time-generalized multivariate analysis of EEG responses reveals a cascading architecture of semantic mismatch processing , 2018, Brain and Language.

[24]  M. Kutas,et al.  An electrophysiological study of scene effects on object identification. , 2003, Brain research. Cognitive brain research.

[25]  Jodi L. Davenport,et al.  Scene Consistency in Object and Background Perception , 2004, Psychological science.

[26]  Marius V. Peelen,et al.  Interaction between Scene and Object Processing Revealed by Human fMRI and MEG Decoding , 2017, The Journal of Neuroscience.

[27]  S. Dehaene,et al.  Characterizing the dynamics of mental representations: the temporal generalization method , 2014, Trends in Cognitive Sciences.

[28]  M. Bar Visual objects in context , 2004, Nature Reviews Neuroscience.

[29]  R. Verleger On the utility of P3 latency as an index of mental chronometry. , 1997, Psychophysiology.

[30]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[31]  L. Deouell,et al.  Synchronous contextual irregularities affect early scene processing: Replication and extension , 2014, Neuropsychologia.

[32]  Bo Ekman In Search of Common Sense , 2007 .

[33]  John M Henderson,et al.  The time course of initial scene processing for eye movement guidance in natural scene search. , 2010, Journal of vision.

[34]  Marta Kutas,et al.  The intractability of scaling scalp distributions to infer neuroelectric sources. , 2002, Psychophysiology.

[35]  Kara D. Federmeier,et al.  Meaning and modality: influences of context, semantic memory organization, and perceptual predictability on picture processing. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[36]  Phillip J. Holcomb,et al.  Two Neurocognitive Mechanisms of Semantic Integration during the Comprehension of Visual Real-world Events , 2008, Journal of Cognitive Neuroscience.

[37]  L. Deouell,et al.  ERP evidence for context congruity effects during simultaneous object–scene processing , 2010, Neuropsychologia.

[38]  M. Bar The proactive brain: using analogies and associations to generate predictions , 2007, Trends in Cognitive Sciences.

[39]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[40]  Tom Eichele,et al.  Semi-automatic identification of independent components representing EEG artifact , 2009, Clinical Neurophysiology.

[41]  J. Wolfe,et al.  Differential Electrophysiological Signatures of Semantic and Syntactic Scene Processing , 2013, Psychological science.

[42]  M. Schlesewsky,et al.  The P600-as-P3 hypothesis revisited: Single-trial analyses reveal that the late EEG positivity following linguistically deviant material is reaction time aligned , 2014, Brain and Language.

[43]  Katherine J. Midgley,et al.  Phonological and semantic priming in American Sign Language: N300 and N400 effects , 2018, Language, cognition and neuroscience.

[44]  Roel M. Willems,et al.  Seeing and Hearing Meaning: ERP and fMRI Evidence of Word versus Picture Integration into a Sentence Context , 2008, Journal of Cognitive Neuroscience.

[45]  R. Gopal,et al.  Electrophysiological investigation , 2018, Medicine.

[46]  J. Wolfe,et al.  The role of memory for visual search in scenes , 2015, Annals of the New York Academy of Sciences.

[47]  et al.,et al.  Jupyter Notebooks - a publishing format for reproducible computational workflows , 2016, ELPUB.

[48]  George L. Malcolm,et al.  ERP correlates of spatially incongruent object identification during scene viewing: Contextual expectancy versus simultaneous processing , 2012, Neuropsychologia.

[49]  Irving Biederman,et al.  On the Semantics of a Glance at a Scene , 2017 .

[50]  Blake W. Johnson,et al.  Comparison of the N300 and N400 ERPs to picture stimuli in congruent and incongruent contexts , 2002, Clinical Neurophysiology.

[51]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[52]  Dejan Draschkow,et al.  Scene grammar shapes the way we interact with objects, strengthens memories, and speeds search , 2017, Scientific Reports.

[53]  P. Holcomb,et al.  An electrophysiological investigation of semantic priming with pictures of real objects. , 1999, Psychophysiology.