Short-term memory stores organized by information domain

Vision and audition have complementary affinities, with vision excelling in spatial resolution and audition excelling in temporal resolution. Here, we investigated the relationships among the visual and auditory modalities and spatial and temporal short-term memory (STM) using change detection tasks. We created short sequences of visual or auditory items, such that each item within a sequence arose at a unique spatial location at a unique time. On each trial, two successive sequences were presented; subjects attended to either space (the sequence of locations) or time (the sequence of inter item intervals) and reported whether the patterns of locations or intervals were identical. Each subject completed blocks of unimodal trials (both sequences presented in the same modality) and crossmodal trials (Sequence 1 visual, Sequence 2 auditory, or vice versa) for both spatial and temporal tasks. We found a strong interaction between modality and task: Spatial performance was best on unimodal visual trials, whereas temporal performance was best on unimodal auditory trials. The order of modalities on crossmodal trials also mattered, suggesting that perceptual fidelity at encoding is critical to STM. Critically, no cost was attributable to crossmodal comparison: In both tasks, performance on crossmodal trials was as good as or better than on the weaker unimodal trials. STM representations of space and time can guide change detection in either the visual or the auditory modality, suggesting that the temporal or spatial organization of STM may supersede sensory-specific organization.

[1]  H. Helson.  THE TAU EFFECT--AN EXAMPLE OF PSYCHOLOGICAL RELATIVITY. , 1931, Science.

[2]  John Cohen,et al.  A New Phenomenon in Time Judgment , 1953, Nature.

[3]  C. V. Jackson,et al.  Visual Factors in Auditory Localization , 1953 .

[4]  D. Hubel,et al.  Receptive fields of single neurones in the cat's striate cortex , 1959, The Journal of physiology.

[5]  C. Douglas Creelman,et al.  Human Discrimination of Auditory Duration , 1962 .

[6]  I. Howard,et al.  Human Spatial Orientation , 1966 .

[7]  T. Parks,et al.  Short-term memory while shadowing: recall of visually and of aurally presented letters. , 1970, Journal of experimental psychology.

[8]  M. Merzenich,et al.  Representation of the cochlear partition of the superior temporal plane of the macaque monkey. , 1973, Brain research.

[9]  W. A. Phillips.  On the distinction between sensory storage and short-term visual memory , 1974 .

[10]  S. Goldstone,et al.  Studies of Auditory-Visual Differences in Human Time Judgment: 2. More Transmitted Information with Sounds than Lights , 1974, Perceptual and motor skills.

[11]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[12]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[13]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[14]  Giuseppe Vallar,et al.  Fractionation of working memory: Neuropsychological evidence for a phonological short-term store , 1984 .

[15]  E. Farmer,et al.  Evidence for a Visuo-Spatial Scratch-Pad in Working Memory , 1986 .

[16]  W. Balch,et al.  The interaction of modality condition and presentation rate in short-term contour recognition , 1986, Perception & psychophysics.

[17]  Alan D. Baddeley,et al.  Effects of Background Music on Phonological Short-Term Memory , 1989 .

[18]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[19]  H. Pashler,et al.  Close binding of identity and location in visual feature perception. , 1990, Journal of experimental psychology. Human perception and performance.

[20]  R H Logie,et al.  Interference with visual short-term memory. , 1990, Acta psychologica.

[21]  J. Richard Hanley,et al.  Impairment of the Visuo-Spatial Sketch Pad , 1991, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[22]  T. Pechmann,et al.  Interference in memory for tonal pitch: Implications for a working-memory model , 1992, Memory & cognition.

[23]  R. Ivry,et al.  Perception and production of temporal intervals across a range of durations: evidence for a common timing mechanism. , 1995, Journal of experimental psychology. Human perception and performance.

[24]  L. Demany,et al.  Speech versus nonspeech in pitch memory. , 1996, The Journal of the Acoustical Society of America.

[25]  A Baddeley,et al.  The fractionation of working memory. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[26]  M. Pitt,et al.  Interference effects in short-term memory for timbre. , 1997, The Journal of the Acoustical Society of America.

[27]  G. Mangun,et al.  Tonotopy in human auditory cortex examined with functional magnetic resonance imaging , 1997, Human brain mapping.

[28]  G. Aschersleben,et al.  Automatic visual bias of perceived auditory location , 1998 .

[29]  D. Geary,et al.  Psychonomic Bulletin Review , 2000 .

[30]  G. Logan,et al.  Modality differences in short-term memory for rhythms , 2000, Memory & cognition.

[31]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[32]  F. Parmentier,et al.  Functional characteristics of auditory temporal-spatial short-term memory: evidence from serial order errors. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[33]  S Martinkauppi,et al.  Working memory of auditory localization. , 2000, Cerebral cortex.

[34]  Robert H. Logie,et al.  Visual Similarity Effects in Immediate Verbal Serial Recall , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[35]  R Fendrich,et al.  The temporal cross-capture of audition and vision , 2001, Perception & psychophysics.

[36]  G. Woodman,et al.  Storage of features, conjunctions and objects in visual working memory. , 2001, Journal of experimental psychology. Human perception and performance.

[37]  Marc W. Howard,et al.  A distributed representation of temporal context , 2002 .

[38]  Alan Baddeley,et al.  Disruption of Verbal Stm by Irrelevant Speech, Articulatory Suppression, and Manual Tapping: Do they have a Common Source? , 2003, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[39]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[40]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[41]  R. Ruibal,et al.  Mapping of Interactions in the Pitch Memory Store , 2004 .

[42]  Elke B. Lange.  Disruption of attention by irrelevant stimuli in serial recall , 2005 .

[43]  Randolph Blake,et al.  Hearing What the Eyes See , 2005, Psychological science.

[44]  Dylan M. Jones,et al.  A spatial modality effect in serial memory. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[45]  Doug J. K. Barrett,et al.  Response preferences for “what” and “where” in human non-primary auditory cortex , 2006, NeuroImage.

[46]  Lotfi B Merabet,et al.  Visual Topography of Human Intraparietal Sulcus , 2007, The Journal of Neuroscience.

[47]  Michael J Kahana,et al.  Auditory Short-Term Memory Behaves Like Visual Short-Term Memory , 2007, PLoS biology.

[48]  N. Cowan,et al.  A central capacity limit to the simultaneous storage of visual and auditory arrays in working memory. , 2007, Journal of experimental psychology. General.

[49]  Hubert D. Zimmer,et al.  Modality and domain specific components in auditory and visual working memory tasks , 2008, Cognitive Processing.

[50]  T. Klingberg,et al.  Prefrontal cortex and basal ganglia control access to working memory , 2008, Nature Neuroscience.

[51]  Jan Theeuwes,et al.  Pip and pop: nonspatial auditory signals improve spatial visual search. , 2008, Journal of experimental psychology. Human perception and performance.

[52]  Sébastien Tremblay,et al.  Revisiting evidence for modularity and functional equivalence across verbal and spatial domains in memory. , 2008, Journal of experimental psychology. Learning, memory, and cognition.

[53]  S. Luck,et al.  Sudden Death and Gradual Decay in Visual Working Memory , 2009, Psychological science.

[54]  C. Summerfield,et al.  Expectation (and attention) in visual cognition , 2009, Trends in Cognitive Sciences.

[55]  Jen-Chuen Hsieh,et al.  Common neural mechanisms for explicit timing in the sub-second range , 2009, Neuroreport.

[56]  D. Burr,et al.  Auditory dominance over vision in the perception of interval duration , 2009, Experimental Brain Research.

[57]  S. Kastner,et al.  Topographic maps in human frontal and parietal cortex , 2009, Trends in Cognitive Sciences.

[58]  T. Alloway,et al.  The effect of articulatory suppression and manual tapping on serial recall , 2010 .

[59]  J. Devin McAuley,et al.  Modality effects in rhythm processing: Auditory encoding of visual rhythms is neither obligatory nor automatic , 2010, Attention, perception & psychophysics.

[60]  Bradley Voytek,et al.  Prefrontal cortex and basal ganglia contributions to visual working memory , 2010, Proceedings of the National Academy of Sciences.

[61]  Richard S. J. Frackowiak,et al.  Human Primary Auditory Cortex Follows the Shape of Heschl's Gyrus , 2011, The Journal of Neuroscience.

[62]  Timothy F. Brady,et al.  A review of visual memory capacity: Beyond individual items and toward structured representations. , 2011, Journal of vision.

[63]  Deborah L. Harrington,et al.  Integrative Neuroscience Original Research Article Neural Underpinnings of Distortions in the Experience of Time across Senses , 2022 .

[64]  Daryl Fougnie,et al.  What limits working memory capacity? Evidence for modality-specific sources to the simultaneous storage of visual and auditory arrays. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[65]  Barbara G Shinn-Cunningham,et al.  Nothing Is Irrelevant in a Noisy World: Sensory Illusions Reveal Obligatory within-and across-Modality Integration , 2012, The Journal of Neuroscience.

[66]  A. Baddeley.  Working memory: theories, models, and controversies. , 2012, Annual review of psychology.

[67]  Frank Tong,et al.  Imagery and visual working memory: one and the same? , 2013, Trends in Cognitive Sciences.

[68]  E. Vogel,et al.  Visual working memory capacity: from psychophysics and neurobiology to individual differences , 2013, Trends in Cognitive Sciences.

[69]  Alan D Baddeley,et al.  Memory for serial order across domains: An overview of the literature and directions for future research. , 2014, Psychological bulletin.

[70]  W. Ma,et al.  Changing concepts of working memory , 2014, Nature Neuroscience.

[71]  Daryl Fougnie,et al.  Working memory storage is intrinsically domain specific. , 2015, Journal of experimental psychology. General.

[72]  Chris Donkin,et al.  Verbal labeling, gradual decay, and sudden death in visual short-term memory , 2015, Psychonomic bulletin & review.

[73]  Robert Sekuler,et al.  Memory and learning with rapid audiovisual sequences. , 2015, Journal of vision.

[74]  Barbara G. Shinn-Cunningham,et al.  Short-Term Memory for Space and Time Flexibly Recruit Complementary Sensory-Biased Frontal Lobe Attention Networks , 2015, Neuron.

[75]  Barbara G Shinn-Cunningham,et al.  Auditory Spatial Coding Flexibly Recruits Anterior, but Not Posterior, Visuotopic Parietal Cortex , 2015, Cerebral cortex.