Behavioral decoding of working memory items inside and outside the focus of attention

How we attend to our thoughts affects how we attend to our environment. Holding information in working memory can automatically bias visual attention toward matching information. By observing attentional biases on reaction times to visual search during a memory delay, it is possible to reconstruct the source of that bias using machine learning techniques and thereby behaviorally decode the content of working memory. Can this be done when more than one item is held in working memory? There is some evidence that multiple items can simultaneously bias attention, but the effects have been inconsistent. One explanation may be that items are stored in different states depending on the current task demands. Recent models propose functionally distinct states of representation for items inside versus outside the focus of attention. Here, we use behavioral decoding to evaluate whether multiple memory items—including temporarily irrelevant items outside the focus of attention—exert biases on visual attention. Only the single item in the focus of attention was decodable. The other item showed a brief attentional bias that dissipated until it returned to the focus of attention. These results support the idea of dynamic, flexible states of working memory across time and priority.

[1]  Valerie M. Beck,et al.  Simultaneous Control of Attention by Multiple Working Memory Representations , 2012, Psychological science.

[2]  Elkan G. Akyürek,et al.  Dynamic hidden states underlying working memory guided behaviour , 2017, Nature Neuroscience.

[3]  L. Chelazzi,et al.  Associative knowledge controls deployment of visual selective attention , 2003, Nature Neuroscience.

[4]  M. Tsodyks,et al.  Working models of working memory , 2014, Current Opinion in Neurobiology.

[5]  P. Roelfsema,et al.  Different States in Visual Working Memory: When It Guides Attention and When It Does Not , 2022 .

[6]  N. Cowan Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information-processing system. , 1988, Psychological bulletin.

[7]  K. Oberauer Access to information in working memory: exploring the focus of attention. , 2002, Journal of experimental psychology. Learning, memory, and cognition.

[8]  G. Woodman,et al.  Enhancing long-term memory with stimulation tunes visual attention in one trial , 2014, Proceedings of the National Academy of Sciences.

[9]  Andrew Hollingworth,et al.  Competition in Saccade Target Selection Reveals Attentional Guidance by Simultaneously Active Working Memory Representations , 2017, Journal of experimental psychology. Human perception and performance.

[10]  M. Stokes ‘Activity-silent’ working memory in prefrontal cortex: a dynamic coding framework , 2015, Trends in Cognitive Sciences.

[11]  Jan Theeuwes,et al.  In competition for the attentional template: can multiple items within visual working memory guide attention? , 2014, Journal of experimental psychology. Human perception and performance.

[12]  Alessandra S. Souza,et al.  Retro-cue benefits in working memory without sustained focal attention , 2014, Memory & cognition.

[13]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[14]  John M. Pearson,et al.  Mind-reading without the scanner: Behavioural decoding of working memory content , 2015 .

[15]  Jeffrey S. Johnson,et al.  Implicit memory influences the allocation of attention in visual cortex , 2007, Psychonomic bulletin & review.

[16]  Christian N L Olivers,et al.  Subtle eye movement metrics reveal task-relevant representations prior to visual search. , 2017, Journal of vision.

[17]  A. Nobre,et al.  Top-down modulation: bridging selective attention and working memory , 2012, Trends in Cognitive Sciences.

[18]  Pia Rotshtein,et al.  Common and distinct neural regions for the guidance of selection by visuoverbal information held in memory: Converging evidence from fMRI and rTMS , 2012, Human brain mapping.

[19]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[20]  Edward Awh,et al.  Spatially Selective Alpha Oscillations Reveal Moment-by-Moment Trade-offs between Working Memory and Attention , 2018, Journal of Cognitive Neuroscience.

[21]  Geoffrey F. Woodman,et al.  Attentional Templates in Visual Working Memory , 2011, The Journal of Neuroscience.

[22]  John Jonides,et al.  Trisecting representational states in short-term memory , 2013, Front. Hum. Neurosci..

[23]  Christian N L Olivers,et al.  What drives memory-driven attentional capture? The effects of memory type, display type, and search type. , 2009, Journal of experimental psychology. Human perception and performance.

[24]  Jeffrey D Schall,et al.  The role of working memory representations in the control of attention. , 2007, Cerebral cortex.

[25]  G. Humphreys,et al.  Early, involuntary top-down guidance of attention from working memory. , 2005, Journal of experimental psychology. Human perception and performance.

[26]  W. Ma,et al.  Changing concepts of working memory , 2014, Nature Neuroscience.

[27]  Martin Eimer,et al.  The Control of Single-color and Multiple-color Visual Search by Attentional Templates in Working Memory and in Long-term Memory , 2016, Journal of Cognitive Neuroscience.

[28]  Julie D. Golomb,et al.  A taxonomy of external and internal attention. , 2011, Annual review of psychology.

[29]  Judith E. Fan,et al.  Incidental biasing of attention from visual long-term memory. , 2016, Journal of experimental psychology. Learning, memory, and cognition.

[30]  N. Cowan Attention and Memory: An Integrated Framework , 1995 .

[31]  P. Downing,et al.  Interactions Between Visual Working Memory and Selective Attention , 2000, Psychological science.

[32]  Eren Gunseli,et al.  Is a search template an ordinary working memory? Comparing electrophysiological markers of working memory maintenance for visual search and recognition , 2014, Neuropsychologia.

[33]  Jonathan S. Cant,et al.  Automatic capture of attention by conceptually generated working memory templates , 2015, Attention, Perception, & Psychophysics.

[34]  Mark G. Stokes,et al.  Top-down visual activity underlying VSTM and preparatory attention , 2011, Neuropsychologia.

[35]  Christopher Kennard,et al.  Early oculomotor capture by new onsets driven by the contents of working memory , 2010, Vision Research.

[36]  Alessandra S. Souza,et al.  In search of the focus of attention in working memory: 13 years of the retro-cue effect , 2016, Attention, perception & psychophysics.

[37]  John Duncan,et al.  Shape-specific preparatory activity mediates attention to targets in human visual cortex , 2009, Proceedings of the National Academy of Sciences.

[38]  Sebastian Schneegans,et al.  Restoration of fMRI Decodability Does Not Imply Latent Working Memory States , 2017, Journal of Cognitive Neuroscience.

[39]  Geoffrey F. Woodman,et al.  When Memory Is Not Enough: Electrophysiological Evidence for Goal-dependent Use of Working Memory Representations in Guiding Visual Attention , 2011, Journal of Cognitive Neuroscience.

[40]  David Soto,et al.  Dynamic states in working memory modulate guidance of visual attention: Evidence from an n-back paradigm , 2015 .

[41]  Paul E. Downing,et al.  Competition in visual working memory for control of search , 2003 .

[42]  G. Woodman,et al.  The role of working memory and long-term memory in visual search , 2006 .

[43]  B. McElree,et al.  Attended and Non-Attended States in Working Memory: Accessing Categorized Structures , 1998 .

[44]  J. Wolfe,et al.  Guided Search 2.0 A revised model of visual search , 1994, Psychonomic bulletin & review.

[45]  Adam C. Riggall,et al.  Reactivation of latent working memories with transcranial magnetic stimulation , 2016, Science.

[46]  Andrew R. A. Conway,et al.  Working memory, attention control, and the N-back task: a question of construct validity. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[47]  M. Tsodyks,et al.  Synaptic Theory of Working Memory , 2008, Science.

[48]  B. Postle,et al.  The cognitive neuroscience of working memory. , 2007, Annual review of psychology.

[49]  Tobias Egner,et al.  Decoding working memory content from attentional biases , 2017, Psychonomic bulletin & review.

[50]  G. Humphreys,et al.  Automatic guidance of attention from working memory , 2008, Trends in Cognitive Sciences.

[51]  V. Lamme,et al.  Forgotten but not gone: Retro-cue costs and benefits in a double-cueing paradigm suggest multiple states in visual short-term memory. , 2015, Journal of experimental psychology. Learning, memory, and cognition.

[52]  Feng Du,et al.  Two visual working memory representations simultaneously control attention , 2017, Scientific Reports.

[53]  P. Roelfsema,et al.  The effect of items in working memory on the deployment of attention and the eyes during visual search. , 2006, Journal of experimental psychology. Human perception and performance.

[54]  Jarrod A. Lewis-Peacock,et al.  Neural Evidence for the Flexible Control of Mental Representations. , 2015, Cerebral cortex.

[55]  Jarrod A. Lewis-Peacock,et al.  Decoding the internal focus of attention , 2012, Neuropsychologia.

[56]  D. Wickens,et al.  Some characteristics of word encoding , 1973, Memory & cognition.

[57]  Nicholas E. Myers,et al.  Revealing hidden states in visual working memory using electroencephalography , 2015, Front. Syst. Neurosci..

[58]  Andrew Hollingworth,et al.  The Architecture of Interaction Between Visual Working Memory and Visual Attention , 2018, Journal of experimental psychology. Human perception and performance.

[59]  Klaus Oberauer,et al.  Neural Evidence for a Distinction between Short-term Memory and the Focus of Attention , 2012, Journal of Cognitive Neuroscience.

[60]  G. Woodman,et al.  Do the contents of visual working memory automatically influence attentional selection during visual search? , 2007, Journal of experimental psychology. Human perception and performance.

[61]  Klaus Oberauer,et al.  Control of the contents of working memory--a comparison of two paradigms and two age groups. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[62]  Anastasia Kiyonaga,et al.  Neural Representation of Working Memory Content Is Modulated by Visual Attentional Demand , 2017, Journal of Cognitive Neuroscience.

[63]  Klaus Oberauer,et al.  Decoding Attended Information in Short-term Memory: An EEG Study , 2013, Journal of Cognitive Neuroscience.

[64]  A. Nobre,et al.  Orienting Attention Based on Long-Term Memory Experience , 2006, Neuron.

[65]  Jan Theeuwes,et al.  Rapid influences of cued visual memories on attentional guidance , 2015, Annals of the New York Academy of Sciences.

[66]  Simon Farrell,et al.  The Playing Field : Findings for Evaluating Hypotheses About WM Capacity , 2016 .

[67]  Eriko Matsumoto,et al.  Cognitive Control of Attentional Guidance by Visual and Verbal Working Memory Representations , 2017 .

[68]  M. Eimer,et al.  On the difference between working memory and attentional set , 2011, Neuropsychologia.

[69]  Edward F. Ester,et al.  Restoring Latent Visual Working Memory Representations in Human Cortex , 2016, Neuron.

[70]  Andrew Hollingworth,et al.  Memory-based attention capture when multiple items are maintained in visual working memory. , 2016, Journal of experimental psychology. Human perception and performance.

[71]  K. Oberauer Removing irrelevant information from working memory: a cognitive aging study with the modified Sternberg task. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[72]  Patryk A. Laurent,et al.  Value-driven attentional capture , 2011, Proceedings of the National Academy of Sciences.

[73]  Chantal E. Stern,et al.  Long-term memory guidance of visuospatial attention in a change-detection paradigm , 2014, Front. Psychol..

[74]  Stephan Lewandowsky,et al.  The removal of information from working memory , 2018, CogSci.

[75]  T. Egner,et al.  Working memory as internal attention: Toward an integrative account of internal and external selection processes , 2012, Psychonomic Bulletin & Review.

[76]  David Soto,et al.  Automatic guidance of visual attention from verbal working memory. , 2007, Journal of experimental psychology. Human perception and performance.

[77]  J. Jonides,et al.  Overlapping mechanisms of attention and spatial working memory , 2001, Trends in Cognitive Sciences.