A multisensory perspective of working memory

Although our sensory experience is mostly multisensory in nature, research on working memory representations has focused mainly on examining the senses in isolation. Results from the multisensory processing literature make it clear that the senses interact on a more intimate manner than previously assumed. These interactions raise questions regarding the manner in which multisensory information is maintained in working memory. We discuss the current status of research on multisensory processing and the implications of these findings on our theoretical understanding of working memory. To do so, we focus on reviewing working memory research conducted from a multisensory perspective, and discuss the relation between working memory, attention, and multisensory processing in the context of the predictive coding framework. We argue that a multisensory approach to the study of working memory is indispensable to achieve a realistic understanding of how working memory processes maintain and manipulate information.

[1]  N. Cowan,et al.  A central capacity limit to the simultaneous storage of visual and auditory arrays in working memory. , 2007, Journal of experimental psychology. General.

[2]  Nelson Cowan,et al.  Central and peripheral components of working memory storage. , 2014, Journal of experimental psychology. General.

[3]  Karl J. Friston The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[4]  Durk Talsma,et al.  Behind the scenes: how visual memory load biases selective attention during processing of visual streams. , 2013, Psychophysiology.

[5]  A. Treisman,et al.  Binding in short-term visual memory. , 2002, Journal of experimental psychology. General.

[6]  I. A. Clark,et al.  Attention Restores Discrete Items to Visual Short-Term Memory , 2013, Psychological science.

[7]  Christopher C. Berger,et al.  Mental Imagery Changes Multisensory Perception , 2013, Current Biology.

[8]  M. Wallace,et al.  Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. , 1993, Journal of neurophysiology.

[9]  Roy Luria,et al.  Visual Short-term Memory Capacity for Simple and Complex Objects , 2010, Journal of Cognitive Neuroscience.

[10]  G. Calvert Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[11]  Daren Zhang,et al.  Binding facilitates attention switching within working memory. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[12]  Maro G. Machizawa,et al.  Neural measures reveal individual differences in controlling access to working memory , 2005, Nature.

[13]  D. Talsma Predictive coding and multisensory integration: an attentional account of the multisensory mind , 2015, Front. Integr. Neurosci..

[14]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[15]  A. Paivio,et al.  Memory for pictures and sounds: independence of auditory and visual codes. , 1994, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[16]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[17]  N. Cowan The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[18]  Christoph M. Michel,et al.  Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging , 2004, NeuroImage.

[19]  A. Baddeley The episodic buffer: a new component of working memory? , 2000, Trends in Cognitive Sciences.

[20]  Micah M. Murray,et al.  Single-trial multisensory memories affect later auditory and visual object discrimination , 2015, Cognition.

[21]  J. Havelka,et al.  Visuospatial bootstrapping: Long-term memory representations are necessary for implicit binding of verbal and visuospatial working memory , 2012, Psychonomic bulletin & review.

[22]  John J. Foxe,et al.  The case for feedforward multisensory convergence during early cortical processing , 2005, Neuroreport.

[23]  Marta Olivetti Belardinelli,et al.  Multisensory integration affects visuo-spatial working memory. , 2011, Journal of experimental psychology. Human perception and performance.

[24]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[25]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[26]  Jan Theeuwes,et al.  Pip and pop: nonspatial auditory signals improve spatial visual search. , 2008, Journal of experimental psychology. Human perception and performance.

[27]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.

[28]  Daryl Fougnie,et al.  What limits working memory capacity? Evidence for modality-specific sources to the simultaneous storage of visual and auditory arrays. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[29]  Gregory McCarthy,et al.  Polysensory interactions along lateral temporal regions evoked by audiovisual speech. , 2003, Cerebral cortex.

[30]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[31]  B. Postle Working memory as an emergent property of the mind and brain , 2006, Neuroscience.

[32]  L. Busse,et al.  The spread of attention across modalities and space in a multisensory object. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[33]  Nelson Cowan,et al.  Remembering complex objects in visual working memory: do capacity limits restrict objects or features? , 2015, Journal of experimental psychology. Learning, memory, and cognition.

[34]  David A. Sabatino Auditory–Visual Integration , 2014 .

[35]  G. Engel,et al.  Neuropsychology , 1994, Schizophrenia Research.

[36]  Christopher D. Chambers,et al.  Current perspectives and methods in studying neural mechanisms of multisensory interactions , 2012, Neuroscience & Biobehavioral Reviews.

[37]  Robert H. Logie,et al.  Visual Similarity Effects in Immediate Verbal Serial Recall , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[38]  B. Postle,et al.  Dissociation of human caudate nucleus activity in spatial and nonspatial working memory: an event-related fMRI study. , 1999, Brain research. Cognitive brain research.

[39]  G. Woodman,et al.  Storage of features, conjunctions and objects in visual working memory. , 2001, Journal of experimental psychology. Human perception and performance.

[40]  Christian N L Olivers,et al.  Interactions between visual working memory and visual attention. , 2008, Frontiers in bioscience : a journal and virtual library.

[41]  A. Damasio,et al.  Remembering when. , 2002, Scientific American.

[42]  Lars Muckli,et al.  Decoding Sound and Imagery Content in Early Visual Cortex , 2014, Current Biology.

[43]  Francisco del Pozo,et al.  Oscillatory activity in prefrontal and posterior regions during implicit letter-location binding , 2010, NeuroImage.

[44]  E. Macaluso,et al.  Multisensory spatial interactions: a window onto functional integration in the human brain , 2005, Trends in Neurosciences.

[45]  A. Baddeley,et al.  Binding in visual working memory: The role of the episodic buffer , 2011, Neuropsychologia.

[46]  Mark C. Detweiler,et al.  A Connectionist/Control Architecture for Working Memory , 1988 .

[47]  A. Yonelinas The hippocampus supports high-resolution binding in the service of perception, working memory and long-term memory , 2013, Behavioural Brain Research.

[48]  Klaus Oberauer,et al.  Visual working memory declines when more features must be remembered for each object , 2013, Memory & cognition.

[49]  Christopher C. Berger,et al.  The Fusion of Mental Imagery and Sensation in the Temporal Association Cortex , 2014, The Journal of Neuroscience.

[50]  J. Driver,et al.  Audiovisual links in exogenous covert spatial orienting , 1997, Perception & psychophysics.

[51]  Daniel Senkowski,et al.  Multisensory processing and oscillatory gamma responses: effects of spatial selective attention , 2005, Experimental Brain Research.

[52]  Brigitte Röder,et al.  Semantic confusion regarding the development of multisensory integration: a practical solution , 2010, The European journal of neuroscience.

[53]  J. Theeuwes,et al.  Auditory and visual capture during focused visual attention. , 2009, Journal of experimental psychology. Human perception and performance.

[54]  Henrik Olsson,et al.  Visual memory needs categories. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[55]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[56]  D. C. Higgins Human Spatial Orientation , 1967, The Yale Journal of Biology and Medicine.

[57]  P. Goolkasian,et al.  Bimodal format effects in working memory. , 2005, The American journal of psychology.

[58]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[59]  M. Murray,et al.  The role of multisensory memories in unisensory object discrimination. , 2005, Brain research. Cognitive brain research.

[60]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[61]  James K. Kroger,et al.  Cross-modal and cross-temporal association in neurons of frontal cortex , 2000, Nature.

[62]  S. Rose Selective attention , 1992, Nature.

[63]  Anthony D Wagner,et al.  Dissociable Effects of Top-Down and Bottom-Up Attention during Episodic Encoding , 2011, The Journal of Neuroscience.

[64]  Nelson Cowan,et al.  When do visual and verbal memories compete , 2004 .

[65]  John J. Foxe,et al.  Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. , 2002, Journal of neurophysiology.

[66]  T. Egner,et al.  Working memory as internal attention: Toward an integrative account of internal and external selection processes , 2012, Psychonomic Bulletin & Review.

[67]  Nelson Cowan,et al.  When visual and verbal memories compete: Evidence of cross-domain limits in working memory , 2004, Psychonomic bulletin & review.

[68]  J. Theeuwes,et al.  Feature-based memory-driven attentional capture: visual working memory content affects visual attention. , 2006, Journal of experimental psychology. Human perception and performance.

[69]  Maro G. Machizawa,et al.  Neural activity predicts individual differences in visual working memory capacity , 2004, Nature.

[70]  E. Vogel,et al.  Interactions between attention and working memory , 2006, Neuroscience.

[71]  M. D’Esposito,et al.  Category-specific modulation of inferior temporal activity during working memory encoding and maintenance. , 2004, Brain research. Cognitive brain research.

[72]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[73]  E. Vogel,et al.  Shape and color conjunction stimuli are represented as bound objects in visual working memory , 2011, Neuropsychologia.

[74]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[75]  A. Capilla,et al.  Temporal dynamics of parietal activity during word-location binding. , 2008, Neuropsychology.

[76]  W. Staines,et al.  Crossmodal influences in somatosensory cortex: Interaction of vision and touch , 2009, Human brain mapping.

[77]  F. Parmentier,et al.  Short article: Is verbal–spatial binding in working memory impaired by a concurrent memory load? , 2009, Quarterly journal of experimental psychology.

[78]  M. Chun,et al.  Organization of visual short-term memory. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[79]  John J. Foxe,et al.  Object‐based attention is multisensory: co‐activation of an object's representations in ignored sensory modalities , 2007, The European journal of neuroscience.

[80]  Jan Theeuwes,et al.  The Attentional Window Modulates Capture by Audiovisual Events , 2012, PloS one.

[81]  Sarah E. MacPherson,et al.  Concurrent performance of two memory tasks: Evidence for domain-specific working memory systems , 2002, Memory & cognition.

[82]  Jan Theeuwes,et al.  Early multisensory interactions affect the competition among multiple visual objects , 2011, NeuroImage.

[83]  王亚周 Involvement of endoplasmic reticulum stress in the necroptosis ofmicroglia/macrophages after spinal cord injury. , 2015 .

[84]  Edward K. Vogel,et al.  The capacity of visual working memory for features and conjunctions , 1997, Nature.

[85]  Marta Olivetti Belardinelli,et al.  Semantic encoding in working memory: Is there a (multi)modality effect? , 2009, Memory.

[86]  Richard C. Atkinson,et al.  Human Memory: A Proposed System and its Control Processes , 1968, Psychology of Learning and Motivation.

[87]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[88]  Katherine Guérard,et al.  Asymmetric binding in serial memory for verbal and spatial information , 2012, Memory & Cognition.

[89]  M. D’Esposito Working memory. , 2008, Handbook of clinical neurology.

[90]  Hans-Jochen Heinze,et al.  Representations in human visual short-term memory: an event-related brain potential study , 1999, Neuroscience Letters.

[91]  J. D. E. Gabrieli,et al.  Integration of diverse information in working memory within the frontal lobe , 2000, Nature Neuroscience.

[92]  Micah M. Murray,et al.  Electrical neuroimaging of memory discrimination based on single-trial multisensory learning , 2012, NeuroImage.

[93]  M. D’Esposito,et al.  Dissecting Contributions of Prefrontal Cortex and Fusiform Face Area to Face Working Memory , 2003, Journal of Cognitive Neuroscience.

[94]  V. Prabhakaran,et al.  The neural correlates of age effects on verbal–spatial binding in working memory , 2014, Behavioural Brain Research.

[95]  Nelson Cowan,et al.  When do visual and verbal memories conflict? The importance of working-memory load and retrieval. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[96]  R. Campbell,et al.  Audiovisual Integration of Speech Falters under High Attention Demands , 2005, Current Biology.

[97]  Salvador Soto-Faraco,et al.  Attention to touch weakens audiovisual speech integration , 2007, Experimental Brain Research.

[98]  K. Oberauer,et al.  Attention to Information in Working Memory , 2012 .

[99]  Marina Schmid,et al.  Imagery And Verbal Processes , 2016 .

[100]  R. Nieuwenhuys The neocortex , 1994, Anatomy and Embryology.

[101]  Ingrid R Olson,et al.  Remembering “what” brings along “where” in visual working memory , 2005, Perception & psychophysics.

[102]  Leo Poom,et al.  Visual working memory capacity and stimulus categories: a behavioral and electrophysiological investigation , 2011, Experimental Brain Research.

[103]  A. Paivio Mental Representations: A Dual Coding Approach , 1986 .

[104]  Simon Farrell,et al.  The Playing Field : Findings for Evaluating Hypotheses About WM Capacity , 2016 .

[105]  J. Driver,et al.  Audiovisual links in endogenous covert spatial attention. , 1996, Journal of experimental psychology. Human perception and performance.

[106]  B. Postle,et al.  The cognitive neuroscience of working memory. , 2007, Annual review of psychology.