Predicting the recognition of natural scenes from single trial MEG recordings of brain activity

In our daily life we look at many scenes. Some are rapidly forgotten, but others we recognize later. We accurately predicted recognition success with natural scene photographs using single trial magnetoencephalography (MEG) measures of brain activation. Specifically, we demonstrate that MEG responses in the initial 600 ms following the onset of scene photographs allow for prediction accuracy rates up to 84.1% using linear Support-Vector-Machine classification (lSVM). A permutation test confirmed that all lSVM based prediction rates were significantly better than "guessing". More generally, we present four approaches to analyzing brain function using lSVMs. (1) We show that lSVMs can be used to extract spatio-temporal patterns of brain activation from MEG-data. (2) We show lSVM classification can demonstrate significant correlations between comparatively early and late processes predictive of scene recognition, indicating dependencies between these processes over time. (3) We use lSVM classification to compare the information content of oscillatory and event-related MEG-activations and show they contain a similar amount of and largely overlapping information. (4) A more detailed analysis of single-trial predictiveness of different frequency bands revealed that theta band activity around 5 Hz allowed for highest prediction rates, and these rates are indistinguishable from those obtained with a full dataset. In sum our results clearly demonstrate that lSVMs can reliably predict natural scene recognition from single trial MEG-activation measures and can be a useful tool for analyzing predictive brain function.

[1]  J. Desmond,et al.  Making memories: brain activity that predicts how well visual experience will be remembered. , 1998, Science.

[2]  Mircea Ariel Schoenfeld,et al.  A multivariate, spatiotemporal analysis of electromagnetic time-frequency data of recognition memory , 2003, NeuroImage.

[3]  John D E Gabrieli,et al.  Neural correlates of actual and predicted memory formation , 2005, Nature Neuroscience.

[4]  P Suppes,et al.  Brain wave recognition of words. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[5]  Janaina Mourão Miranda,et al.  Classifying brain states and determining the discriminating activation patterns: Support Vector Machine on functional MRI data , 2005, NeuroImage.

[6]  G Kovács,et al.  Cortical correlate of pattern backward masking. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[7]  Mark D'Esposito,et al.  From cognitive to neural models of working memory , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[8]  J. I The Design of Experiments , 1936, Nature.

[9]  D. Schacter,et al.  When encoding yields remembering: insights from event-related neuroimaging. , 1999, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[10]  Mieke Verfaellie,et al.  The Role of VMPC in Metamemorial Judgments of Content Retrievability , 2005, Journal of Cognitive Neuroscience.

[11]  Hans-Jochen Heinze,et al.  Speed limits 2 Running head : SPEED LIMITS IN NATURAL SCENE DISCRIMINATION Speed Limits : Orientation and Semantic Context Interactions Constrain Natural Scene Discrimination Dynamics , 2007 .

[12]  H. Heinze,et al.  The oscillatory dynamics of recognition memory and its relationship to event-related responses. , 2005, Cerebral cortex.

[13]  P. Good Permutation, Parametric, and Bootstrap Tests of Hypotheses , 2005 .

[14]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[15]  P. Glimcher Indeterminacy in brain and behavior. , 2005, Annual review of psychology.

[16]  W. Klimesch,et al.  Theta oscillations and the ERP old/new effect: independent phenomena? , 2000, Clinical Neurophysiology.

[17]  P. Glimcher,et al.  Activity in Posterior Parietal Cortex Is Correlated with the Relative Subjective Desirability of Action , 2004, Neuron.

[18]  R. Passingham,et al.  Reading Hidden Intentions in the Human Brain , 2007, Current Biology.

[19]  J. Rieger,et al.  Sensory and cognitive contributions of color to the recognition of natural scenes , 2000, Current Biology.

[20]  B. Oken Electrophysiology of Mind: Event-Related Brain Potentials and Cognition , 1996 .

[21]  K. Paller,et al.  Observing the transformation of experience into memory , 2002, Trends in Cognitive Sciences.

[22]  T. Allison,et al.  Electrophysiological studies of human face perception. I: Potentials generated in occipitotemporal cortex by face and non-face stimuli. , 1999, Cerebral cortex.

[23]  L. Standing Learning 10,000 pictures. , 1973, The Quarterly journal of experimental psychology.

[24]  Daniel L. Schacter,et al.  When encoding yields remembering , 1999 .

[25]  N. Birbaumer,et al.  A brain–computer interface (BCI) for the locked-in: comparison of different EEG classifications for the thought translation device , 2003, Clinical Neurophysiology.

[26]  D Friedman,et al.  Event‐related potential (ERP) studies of memory encoding and retrieval: A selective review , 2000, Microscopy research and technique.

[27]  K. Grill-Spector,et al.  The dynamics of object-selective activation correlate with recognition performance in humans , 2000, Nature Neuroscience.

[28]  M. Kutas,et al.  Neural correlates of encoding in an incidental learning paradigm. , 1987, Electroencephalography and clinical neurophysiology.

[29]  M Doppelmayr,et al.  Brain oscillations and human memory: EEG correlates in the upper alpha and theta band , 1997, Neuroscience Letters.

[30]  S. Thorpe,et al.  Speed of processing in the human visual system , 1996, Nature.

[31]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[32]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[33]  Dr. Robert Miller Cortico-Hippocampal Interplay and the Representation of Contexts in the Brain , 1991, Studies of Brain Function.

[34]  W. Freeman Mass action in the nervous system : examination of the neurophysiological basis of adaptive behavior through the EEG , 1975 .

[35]  S. Edelman,et al.  Human Brain Mapping 6:316–328(1998) � A Sequence of Object-Processing Stages Revealed by fMRI in the Human Occipital Lobe , 2022 .

[36]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[37]  Ole Jensen,et al.  Reading the hippocampal code by theta phase-locking , 2005, Trends in Cognitive Sciences.

[38]  G. Rees,et al.  Predicting the orientation of invisible stimuli from activity in human primary visual cortex , 2005, Nature Neuroscience.

[39]  Marc W Howard,et al.  Theta and Gamma Oscillations during Encoding Predict Subsequent Recall , 2003, The Journal of Neuroscience.

[40]  Thorsten Joachims,et al.  Learning to classify text using support vector machines - methods, theory and algorithms , 2002, The Kluwer international series in engineering and computer science.

[41]  J. Movshon,et al.  A computational analysis of the relationship between neuronal and behavioral responses to visual motion , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[42]  L. Standing Learning 10000 pictures , 1973 .

[43]  Elmer S. West From the U. S. A. , 1965 .

[44]  M. Tovée,et al.  Processing speed in the cerebral cortex and the neurophysiology of visual masking , 1994, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[45]  D. Jeffreys Evoked Potential Studies of Face and Object Processing , 1996 .

[46]  Patrick Suppes,et al.  Single-Trial Classification of MEG Recordings , 2007, IEEE Transactions on Biomedical Engineering.

[47]  G Buzsáki,et al.  The hippocampo-neocortical dialogue. , 1996, Cerebral cortex.

[48]  R. Nickerson,et al.  SHORT-TERM MEMORY FOR COMPLEX MEANINGFUL VISUAL CONFIGURATIONS: A DEMONSTRATION OF CAPACITY. , 1965, Canadian journal of psychology.

[49]  R. Oostenveld,et al.  Theta and Gamma Oscillations Predict Encoding and Retrieval of Declarative Memory , 2006, The Journal of Neuroscience.

[50]  Robert A. Lordo,et al.  Learning from Data: Concepts, Theory, and Methods , 2001, Technometrics.

[51]  W. Klimesch,et al.  Theta band power in the human scalp EEG and the encoding of new information , 1996, Neuroreport.

[52]  David G. Stork,et al.  Pattern Classification , 1973 .

[53]  E. Basar Memory as the "whole brain work": a large-scale model based on "oscillations in super-synergy". , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[54]  Karl R Gegenfurtner,et al.  The dynamics of visual pattern masking in natural scene processing: a magnetoencephalography study. , 2005, Journal of vision.

[55]  T. Sejnowski,et al.  Dynamic Brain Sources of Visual Evoked Responses , 2002, Science.

[56]  J. Martinerie,et al.  The brainweb: Phase synchronization and large-scale integration , 2001, Nature Reviews Neuroscience.

[57]  R. Oostenveld,et al.  Successful declarative memory formation is associated with ongoing activity during encoding in a distributed neocortical network related to working memory: A magnetoencephalography study , 2006, Neuroscience.

[58]  L. M. Ward,et al.  Synchronous neural oscillations and cognitive processes , 2003, Trends in Cognitive Sciences.