Decoding oscillatory representations of visual stimuli in episodic memory and working memory

Theories inspired by electrophysiological studies in animals suggest that the replay of past experiences plays an important role in episodic memory as well as working memory; yet very little is known about the neural characteristics of such replay in the human brain. This thesis consists of neuroimaging experiments for studying the temporal characteristics of the replay in the human brain and analytical methods for decoding replay. To that end, oscillatory neural activity patterns were recorded from healthy young adults via a non-invasive electrophysiological technique (Magnetoencephalography, MEG). Firstly, a pipeline for decoding MEG data using machine learning algorithms was eveloped and proposed. Then using an associative recognition experiment, we marked the neural signature for categorical visual information (about faces and scenes) during encoding. These markers of encoding-related experiences were then used for detecting the replay during retrieval - triggered by an associative memory cue. As a result, replay was detected at about 500 ms from onset of the cue. The results suggest that episodic recollection and replay are accomplished within 500 ms. Next, in a working memory experiment, I used item speci c visual information for tracking the replay of oscillatory activity while maintaining that information. Three visual stimuli with presumably distinct cortical representations were selected (types: a face, a banana, and a chair) and presented in a sequential order. Event-related responses during encoding showed a main e ect of item type and working memory load at 400-500 ms from onset of the stimuli. Using a decoding approach, it was possible to categorize oscillatory patterns related to each of the three stimulus types. These decoders are now used as markers of item speci c replay in working memory during the maintenance phase. This analysis is ongoing. Finally, we proposed a pipeline for detecting an optimal feature space for decoding MEG data at a group level because the previous pipeline relied on di erent features across subjects for decoding. Here the Canonical Variates Analysis of beamformer reconstructed MEG data in source space was used. Canonical Variates Analysis stimated the dependency of the selected features of MEG data to the experimental conditions and enabled multivariate decoding of MEG signal in the source space. Thus this proposed method was an optimal way for group level inference of MEG multivariate analysis. Overall, the MEG based decoding of the representation of visual stimuli was shown in source and sensor spaces. Also, our results revealed the temporal characteristic of replay in an episodic memory experiment.

[1]  C. Colby.  Action-Oriented Spatial Reference Frames in Cortex , 1998, Neuron.

[2]  M. Hasselmo,et al.  Opinion TRENDS in Cognitive Sciences Vol.10 No.11 Mechanisms underlying working memory for novel information , 2022 .

[3]  K. Norman.  How hippocampus and cortex contribute to recognition memory: Revisiting the complementary learning systems model , 2010, Hippocampus.

[4]  Michel Besserve,et al.  Classification methods for ongoing EEG and MEG signals. , 2007, Biological research.

[5]  Margaret F. Carr,et al.  Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval , 2011, Nature Neuroscience.

[6]  M. Hasselmo,et al.  Graded persistent activity in entorhinal cortex neurons , 2002, Nature.

[7]  D. Bruce,et al.  Fifty Years Since Lashley's In Search of the Engram: Refutations and Conjectures , 2001, Journal of the history of the neurosciences.

[8]  Sidney R. Lehky,et al.  Frontiers in Computational Neuroscience Computational Neuroscience , 2022 .

[9]  E. Vogel,et al.  Interactions between attention and working memory , 2006, Neuroscience.

[10]  M. Mishkin.  A memory system in the monkey. , 1982, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[11]  G. Barnes,et al.  Identifying spatially overlapping local cortical networks with MEG , 2009, Human brain mapping.

[12]  J. Fell,et al.  Rhinal–hippocampal theta coherence during declarative memory formation: interaction with gamma synchronization? , 2003, The European journal of neuroscience.

[13]  E. Tulving,et al.  Organization of memory. , 1973 .

[14]  A. Dale,et al.  Building memories: remembering and forgetting of verbal experiences as predicted by brain activity. , 1998, Science.

[15]  David J. Freedman,et al.  Representation of the Quantity of Visual Items in the Primate Prefrontal Cortex , 2002, Science.

[16]  T. Allison,et al.  Electrophysiological studies of human face perception. III: Effects of top-down processing on face-specific potentials. , 1999, Cerebral cortex.

[17]  Chris J. Tinsley,et al.  Selective representation of task‐relevant objects and locations in the monkey prefrontal cortex , 2006, The European journal of neuroscience.

[18]  M. Manosevitz.  High-Speed Scanning in Human Memory , .

[19]  Y. Okada,et al.  Contributions of principal neocortical neurons to magnetoencephalography and electroencephalography signals , 2006, The Journal of physiology.

[20]  Charan Ranganath,et al.  Medial Temporal Lobe Activity Predicts Successful Relational Memory Binding , 2008, The Journal of Neuroscience.

[21]  Russell A. Poldrack,et al.  Putting names to faces: Successful encoding of associative memories activates the anterior hippocampal formation , 2003, NeuroImage.

[22]  F. Craik.  Levels of processing: Past, present... and future? , 2002, Memory.

[23]  J. Pernier,et al.  Induced gamma-band activity during the delay of a visual short-term memory task in humans. , 1998, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[24]  M Mishkin,et al.  Brain activity evidence for recognition without recollection after early hippocampal damage , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[25]  R. Dolan,et al.  Dissociable Temporal Lobe Activations during Emotional Episodic Memory Retrieval , 2000, NeuroImage.

[26]  M. Rugg,et al.  Event-related potentials and recognition memory , 2007, Trends in Cognitive Sciences.

[27]  Anjan Chatterjee,et al.  Visual Working Memory Is Impaired when the Medial Temporal Lobe Is Damaged , 2006, Journal of Cognitive Neuroscience.

[28]  Earl K. Miller,et al.  Selective representation of relevant information by neurons in the primate prefrontal cortex , 1998, Nature.

[29]  F. Craik,et al.  Levels of Pro-cessing: A Framework for Memory Research , 1975 .

[30]  A. J. Collins,et al.  Introduction To Multivariate Analysis , 1981 .

[31]  L R Squire,et al.  Remembering and knowing: two different expressions of declarative memory. , 1995, Journal of experimental psychology. Learning, memory, and cognition.

[32]  David Poeppel,et al.  Application of an MEG eigenspace beamformer to reconstructing spatio‐temporal activities of neural sources , 2002, Human brain mapping.

[33]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[34]  Dwight J. Kravitz,et al.  High-level visual object representations are constrained by position. , 2010, Cerebral cortex.

[35]  M. Rugg,et al.  Separating the Brain Regions Involved in Recollection and Familiarity in Recognition Memory , 2005, The Journal of Neuroscience.

[36]  David E. J. Linden,et al.  Working Memory Load for Faces Modulates P300, N170, and N250r , 2008, Journal of Cognitive Neuroscience.

[37]  Jeremy R. Reynolds,et al.  Developing PFC representations using reinforcement learning , 2009, Cognition.

[38]  T W Picton,et al.  Comparative electrophysiological and hemodynamic measures of neural activation during memory‐retrieval , 2001, Human brain mapping.

[39]  C. Stern,et al.  Medial temporal and prefrontal contributions to working memory tasks with novel and familiar stimuli , 2001, Hippocampus.

[40]  Barbara H. Basden,et al.  Learning about memory. , 1998 .

[41]  Karl J. Friston,et al.  Applications of random field theory to electrophysiology , 2005, Neuroscience Letters.

[42]  Sean M. Polyn,et al.  A context maintenance and retrieval model of organizational processes in free recall. , 2009, Psychological review.

[43]  Brice A. Kuhl,et al.  Multi-voxel patterns of visual category representation during episodic encoding are predictive of subsequent memory , 2012, Neuropsychologia.

[44]  Hiroki R. Hayama,et al.  Electrophysiological dissociation of the neural correlates of recollection and familiarity , 2006, Brain Research.

[45]  N. Kanwisher,et al.  Brain Imaging , 2003, Encyclopedia of Behavioral Medicine.

[46]  B. McNaughton,et al.  Memory trace reactivation in hippocampal and neocortical neuronal ensembles , 2000, Current Opinion in Neurobiology.

[47]  Gabriel Curio,et al.  MEG/EEG sources of the 170-ms response to faces are co-localized in the fusiform gyrus , 2007, NeuroImage.

[48]  P Rappelsberger,et al.  Theta synchronization predicts efficient memory encoding of concrete and abstract nouns , 2000, Neuroreport.

[49]  O. Jensen,et al.  Frontal theta activity in humans increases with memory load in a working memory task , 2002, The European journal of neuroscience.

[50]  J. Fell,et al.  The role of phase synchronization in memory processes , 2011, Nature Reviews Neuroscience.

[51]  R. O’Reilly,et al.  Computational principles of learning in the neocortex and hippocampus , 2000, Hippocampus.

[52]  W. Scoville,et al.  LOSS OF RECENT MEMORY AFTER BILATERAL HIPPOCAMPAL LESIONS , 1957, Journal of neurology, neurosurgery, and psychiatry.

[53]  Chong Wang,et al.  Variational Bayesian Approach to Canonical Correlation Analysis , 2007, IEEE Transactions on Neural Networks.

[54]  C. Mann,et al.  A Practical Treatise on Diseases of the Skin , 1889, Atlanta Medical and Surgical Journal (1884).

[55]  Edward E. Smith,et al.  Temporal dynamics of brain activation during a working memory task , 1997, Nature.

[56]  Albert K. Lee,et al.  Memory of Sequential Experience in the Hippocampus during Slow Wave Sleep , 2002, Neuron.

[57]  M. Mishkin,et al.  Differential effects of early hippocampal pathology on episodic and semantic memory. , 1997, Science.

[58]  F. Tong,et al.  Decoding reveals the contents of visual working memory in early visual areas , 2009, Nature.

[59]  K. Deisseroth,et al.  Optogenetic stimulation of a hippocampal engram activates fear memory recall , 2012, Nature.

[60]  N. Burgess,et al.  The hippocampus is required for short‐term topographical memory in humans , 2007, Hippocampus.

[61]  B. Rossion,et al.  ERP evidence for the speed of face categorization in the human brain: Disentangling the contribution of low-level visual cues from face perception , 2011, Vision Research.

[62]  R. Buckner,et al.  Dissociating State and Item Components of Recognition Memory Using fMRI , 2001, NeuroImage.

[63]  J W Belliveau,et al.  Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. , 1995, Science.

[64]  S. Hanslmayr,et al.  Oscillatory power decreases and long-term memory: the information via desynchronization hypothesis , 2012, Front. Hum. Neurosci..

[65]  W. Klimesch,et al.  Episodic and semantic memory: an analysis in the EEG theta and alpha band. , 1994, Electroencephalography and clinical neurophysiology.

[66]  W. Klimesch,et al.  Theta oscillations and the ERP old/new effect: independent phenomena? , 2000, Clinical Neurophysiology.

[67]  O. Bertrand,et al.  Oscillatory gamma-band (30-70 Hz) activity induced by a visual search task in humans. , 1997, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[68]  R. Passingham,et al.  Reading Hidden Intentions in the Human Brain , 2007, Current Biology.

[69]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[70]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[71]  D. Schacter,et al.  Implicit and explicit memory for new associations in normal and amnesic subjects. , 1985, Journal of experimental psychology. Learning, memory, and cognition.

[72]  Christopher Bishop,et al.  Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics , 2003 .

[73]  M. W. Brown,et al.  Episodic memory, amnesia, and the hippocampal–anterior thalamic axis , 1999, Behavioral and Brain Sciences.

[74]  Leslie G. Ungerleider,et al.  Dissociation of object and spatial visual processing pathways in human extrastriate cortex. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[75]  Y. Yamaguchi,et al.  Neuronal ensemble for visual working memory via interplay of slow and fast oscillations , 2011, The European journal of neuroscience.

[76]  R. Clark,et al.  The medial temporal lobe. , 2004, Annual review of neuroscience.

[77]  D Marr,et al.  Simple memory: a theory for archicortex. , 1971, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[78]  A. Hupbach,et al.  Memory formation, consolidation and transformation , 2012, Neuroscience & Biobehavioral Reviews.

[79]  Michael J. Frank,et al.  Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia , 2006, Neural Computation.

[80]  O. Bertrand,et al.  Oscillatory gamma activity in humans and its role in object representation , 1999, Trends in Cognitive Sciences.

[81]  Michael D. Rugg,et al.  An electrophysiological comparison of two indices of recollection , 1998 .

[82]  Ehren L. Newman,et al.  Moderate excitation leads to weakening of perceptual representations. , 2010, Cerebral cortex.

[83]  William D. Penny,et al.  Bayesian neural networks for classification: how useful is the evidence framework? , 1999, Neural Networks.

[84]  L. Saksida,et al.  The representational–hierarchical view of amnesia: Translation from animal to human , 2010, Neuropsychologia.

[85]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[86]  C. Koch,et al.  Imagery neurons in the human brain , 2000, Nature.

[87]  C. Koch,et al.  Invariant visual representation by single neurons in the human brain , 2005, Nature.

[88]  Fernando H. Lopes da Silva,et al.  Event-related neural activities: what about phase? , 2006 .

[89]  G. Winocur,et al.  Functional neuroanatomy of remote episodic, semantic and spatial memory: a unified account based on multiple trace theory , 2005, Journal of anatomy.

[90]  P. Downing,et al.  Selectivity for the human body in the fusiform gyrus. , 2005, Journal of neurophysiology.

[91]  G. Buzsáki,et al.  Neuronal Oscillations in Cortical Networks , 2004, Science.

[92]  H. Kucera,et al.  Computational analysis of present-day American English , 1967 .

[93]  P. Downing,et al.  Interactions Between Visual Working Memory and Selective Attention , 2000, Psychological science.

[94]  Marcel van Gerven,et al.  MEG-based decoding of the spatiotemporal dynamics of visual category perception , 2013, NeuroImage.

[95]  L. Davachi,et al.  What Constitutes an Episode in Episodic Memory? , 2011, Psychological science.

[96]  H. Eichenbaum,et al.  Hippocampal “Time Cells” Bridge the Gap in Memory for Discontiguous Events , 2011, Neuron.

[97]  William H. Press,et al.  Numerical recipes in C , 2002 .

[98]  Marc W Howard,et al.  Theta and Gamma Oscillations during Encoding Predict Subsequent Recall , 2003, The Journal of Neuroscience.

[99]  R. Oostenveld,et al.  Theta and Gamma Oscillations Predict Encoding and Retrieval of Declarative Memory , 2006, The Journal of Neuroscience.

[100]  Charan Ranganath,et al.  Effects of Unilateral Prefrontal Lesions on Familiarity, Recollection, and Source Memory , 2005, The Journal of Neuroscience.

[101]  David I. Donaldson,et al.  Remember the Source: Dissociating Frontal and Parietal Contributions to Episodic Memory , 2010, Journal of Cognitive Neuroscience.

[102]  M. Moscovitch,et al.  Top-down and bottom-up attention to memory: A hypothesis (AtoM) on the role of the posterior parietal cortex in memory retrieval , 2008, Neuropsychologia.

[103]  Guillaume Flandin,et al.  Bayesian Analysis of fMRI data with Spatial Priors , 2005 .

[104]  Emrah Düzel,et al.  Hippocampal Theta-Phase Modulation of Replay Correlates with Configural-Relational Short-Term Memory Performance , 2011, The Journal of Neuroscience.

[105]  Audrey Duarte,et al.  Medial prefrontal cortex supports source memory accuracy for self-referenced items , 2012, Social neuroscience.

[106]  H. J. Gamble.  Trends in Neurosciences , 1980 .

[107]  Brooke L. Fridley,et al.  Comparison of penalty functions for sparse canonical correlation analysis , 2012, Comput. Stat. Data Anal..

[108]  T. Allison,et al.  Electrophysiological studies of human face perception. II: Response properties of face-specific potentials generated in occipitotemporal cortex. , 1999, Cerebral cortex.

[109]  Joseph Dien,et al.  Separating the visual sentence N400 effect from the P400 sequential expectancy effect: Cognitive and neuroanatomical implications , 2010, Brain Research.

[110]  E. Tulving.  How many memory systems are there , 1985 .

[111]  A. Bartels,et al.  Parietal Cortex Codes for Egocentric Space beyond the Field of View , 2012, Current Biology.

[112]  M. Kahana,et al.  Human hippocampal theta oscillations and the formation of episodic memories , 2012, Hippocampus.

[113]  J. Lisman,et al.  Oscillations in the alpha band (9-12 Hz) increase with memory load during retention in a short-term memory task. , 2002, Cerebral cortex.

[114]  C. Elger,et al.  Human memory formation is accompanied by rhinal–hippocampal coupling and decoupling , 2001, Nature Neuroscience.

[115]  C. Koch,et al.  Category-specific visual responses of single neurons in the human medial temporal lobe , 2000, Nature Neuroscience.

[116]  Ken A. Paller,et al.  Remembering and knowing: Electrophysiological distinctions at encoding but not retrieval , 2009, NeuroImage.

[117]  R. Morris,et al.  Memory Reconsolidation: Sensitivity of Spatial Memory to Inhibition of Protein Synthesis in Dorsal Hippocampus during Encoding and Retrieval , 2006, Neuron.

[118]  R. Kass,et al.  Decoding and cortical source localization for intended movement direction with MEG. , 2010, Journal of neurophysiology.

[119]  H. Heinze,et al.  Electromagnetic Correlates of Recognition Memory Processes , 2006, Clinical EEG and neuroscience.

[120]  Erik A. Wing,et al.  Neural similarity between encoding and retrieval is related to memory via hippocampal interactions. , 2013, Cerebral cortex.

[121]  M. Goldberg,et al.  Space and attention in parietal cortex. , 1999, Annual review of neuroscience.

[122]  O. Jensen,et al.  Cross-frequency coupling between neuronal oscillations , 2007, Trends in Cognitive Sciences.

[123]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[124]  B L McNaughton,et al.  Coordinated Reactivation of Distributed Memory Traces in Primate Neocortex , 2002, Science.

[125]  Jared F. Danker,et al.  The ghosts of brain states past: remembering reactivates the brain regions engaged during encoding. , 2010, Psychological bulletin.

[126]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[127]  David G. Gadian,et al.  A Rapid, Hippocampus-Dependent, Item-Memory Signal that Initiates Context Memory in Humans , 2012, Current Biology.

[128]  Edward E. Smith,et al.  A parametric study of prefrontal cortex involvement in human working memory , 1996, NeuroImage.

[129]  Shannon Tubridy,et al.  Medial temporal lobe contributions to episodic sequence encoding. , 2011, Cerebral cortex.

[130]  L. Nadel,et al.  Update on Memory Systems and Processes , 2011, Neuropsychopharmacology.

[131]  R. Verleger.  Event-related potentials and cognition: A critique of the context updating hypothesis and an alternative interpretation of P3 , 1988, Behavioral and Brain Sciences.

[132]  Y. Dudai.  The restless engram: consolidations never end. , 2012, Annual review of neuroscience.

[133]  Roberto Cabeza,et al.  Parietal Lobe and Episodic Memory: Bilateral Damage Causes Impaired Free Recall of Autobiographical Memory , 2007, The Journal of Neuroscience.

[134]  Scott Makeig,et al.  Information-based modeling of event-related brain dynamics. , 2006, Progress in brain research.

[135]  D. C. Mccarthy,et al.  Hippocampal and neocortical gamma oscillations predict memory formation in humans. , 2006, Cerebral cortex.

[136]  Markus Siegel,et al.  Phase-dependent neuronal coding of objects in short-term memory , 2009, Proceedings of the National Academy of Sciences.

[137]  R. Leahy,et al.  EEG and MEG: forward solutions for inverse methods , 1999, IEEE Transactions on Biomedical Engineering.

[138]  J. Fell,et al.  Phase/amplitude reset and theta–gamma interaction in the human medial temporal lobe during a continuous word recognition memory task , 2005, Hippocampus.

[139]  Vaidehi S. Natu,et al.  Category-Specific Cortical Activity Precedes Retrieval During Memory Search , 2005, Science.

[140]  Karl J. Friston,et al.  Controlling false positive rates in mass-multivariate tests for electromagnetic responses , 2011, NeuroImage.

[141]  A Gevins,et al.  Dynamic cortical networks of verbal and spatial working memory: effects of memory load and task practice. , 1998, Cerebral cortex.

[142]  Robert T. Knight,et al.  Effects of extensive temporal lobe damage or mild hypoxia on recollection and familiarity , 2002, Nature Neuroscience.

[143]  E. Rolls,et al.  Computational analysis of the role of the hippocampus in memory , 1994, Hippocampus.

[144]  H. Heinze,et al.  The oscillatory dynamics of recognition memory and its relationship to event-related responses. , 2005, Cerebral cortex.

[145]  Joachim M. Buhmann,et al.  Bayesian mixed-effects inference on classification performance in hierarchical data sets , 2012, J. Mach. Learn. Res..

[146]  Michael D. Rugg,et al.  Event-related potentials and the recollection of low and high frequency words , 1995, Neuropsychologia.

[147]  Gabriel Curio,et al.  Current multipole expansion to estimate lateral extent of neuronal activity: a theoretical analysis , 2000, IEEE Trans. Biomed. Eng..

[148]  Mircea Ariel Schoenfeld,et al.  A multivariate, spatiotemporal analysis of electromagnetic time-frequency data of recognition memory , 2003, NeuroImage.

[149]  E. Donchin.  Presidential address, 1980. Surprise!...Surprise? , 1981, Psychophysiology.

[150]  David J. Foster,et al.  Reverse replay of behavioural sequences in hippocampal place cells during the awake state , 2006, Nature.

[151]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[152]  M. Moscovitch,et al.  The parietal cortex and episodic memory: an attentional account , 2008, Nature Reviews Neuroscience.

[153]  Xuming He,et al.  Dimension reduction based on constrained canonical correlation and variable filtering , 2008, 0808.0977.

[154]  K. Bäuml,et al.  Brain oscillations dissociate between semantic and nonsemantic encoding of episodic memories. , 2009, Cerebral cortex.

[155]  R. Morris,et al.  Elements of a neurobiological theory of hippocampal function: the role of synaptic plasticity, synaptic tagging and schemas , 2006, The European journal of neuroscience.

[156]  T. Yoshimoto,et al.  Recent Advances in Biomagnetism , 2007 .

[157]  R. Bjork,et al.  Recency-sensitive retrieval processes in long-term free recall☆ , 1974 .

[158]  Karl J. Friston,et al.  EEG and MEG Data Analysis in SPM8 , 2011, Comput. Intell. Neurosci..

[159]  Sean M. Polyn,et al.  The Cambridge Handbook of Computational Psychology: Computational Models of Episodic Memory , 2008 .

[160]  Brice A. Kuhl,et al.  Fidelity of neural reactivation reveals competition between memories , 2011, Proceedings of the National Academy of Sciences.