Mnemonic effects of action simulation from pictures and phrases.

Several theoretical approaches suggest that language comprehension and action observation rely on similar mental simulations. Granted that these two simulations partially overlap, we assumed that simulations stemming from action observations are more direct than those stemming from action phrases. The implied prediction was that simulation from action observation should prevail on simulation from action phrases when their effects are contrasted. The results of three experiments confirmed that, when at encoding the phrases were paired with pictures of actions whose kinematics was incongruent with the implied kinematics of the actions described in the phrases, memory for action phrases was impaired (Experiment 1). However, the reverse was not true: when the pictures were paired with phrases representing actions whose kinematics were incongruent with the kinematics of the actions portrayed in the pictures, memory for pictures portraying actions was not impaired (Experiment 2). Also, in line with evidence that simulations from action phrases and those from action observation partially overlap, when their effects were not contrasted their products were misrecognized. In our experiments, when action phrases only presented at recognition described actions depicted in pictures seen at encoding, they were misrecognized as had already been read at encoding (Experiment 1); further, when pictures only presented at recognition portrayed actions described in phrases presented at encoding, they were misrecognized as seen at encoding (Experiment 2). A third experiment excluded the possibility that the pattern of findings was simply a consequence of better memory for pictures of actions as opposed to memory for action phrases (Experiment 3). The implications of our results in relation to the literature on simulation in language comprehension and action observation are discussed.

[1]  R. Ellis,et al.  Action priming by briefly presented objects. , 2004, Acta psychologica.

[2]  Wolfgang Prinz,et al.  Prospective coding in event representation , 2007, Cognitive Processing.

[3]  Frank Van Overwalle,et al.  Understanding others' actions and goals by mirror and mentalizing systems: A meta-analysis , 2009, NeuroImage.

[4]  Jeffrey N. Rouder,et al.  Bayesian t tests for accepting and rejecting the null hypothesis , 2009, Psychonomic bulletin & review.

[5]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[6]  V. Gallese Before and below ‘theory of mind’: embodied simulation and the neural correlates of social cognition , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[7]  N. Kanwisher,et al.  Activation in Human MT/MST by Static Images with Implied Motion , 2000, Journal of Cognitive Neuroscience.

[8]  Patric Bach,et al.  The affordance-matching hypothesis: how objects guide action understanding and prediction , 2014, Front. Hum. Neurosci..

[9]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[10]  M. Jeannerod The representing brain: Neural correlates of motor intention and imagery , 1994, Behavioral and Brain Sciences.

[11]  V. Gallese,et al.  Processing of Hand-Related Verbs Specifically Affects the Planning and Execution of Arm Reaching Movements , 2012, PloS one.

[12]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[13]  Pierre Feyereisen,et al.  Further investigation on the mnemonic effect of gestures: Their meaning matters , 2006 .

[14]  M Hegarty,et al.  Mental animation in the visuospatial sketchpad: Evidence from dual-task studies , 1997, Memory & cognition.

[15]  G. Knoblich,et al.  The case for motor involvement in perceiving conspecifics. , 2005, Psychological bulletin.

[16]  Volker Haarslev,et al.  Theory and Application of Diagrams , 2003, Lecture Notes in Computer Science.

[17]  Linda Drijvers,et al.  Hearing and seeing meaning in noise: Alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension , 2018, Human brain mapping.

[18]  Giorgia Committeri,et al.  Ready Both to Your and to My Hands: Mapping the Action Space of Others , 2011, PloS one.

[19]  Jean McKendree,et al.  The Role of Representation in Teaching and Learning Critical Thinking , 2002 .

[20]  G. Rizzolatti The mirror neuron system and its function in humans , 2005, Anatomy and Embryology.

[21]  P. Johnson-Laird How We Reason , 2006 .

[22]  Tad T. Brunyé,et al.  Simulating an enactment effect: Pronouns guide action simulation during narrative comprehension , 2010, Cognition.

[23]  A. Glenberg,et al.  Comprehension of illustrated text: Pictures help to build mental models☆ , 1992 .

[24]  Koen Lamberts,et al.  The encoding–retrieval relationship: retrieval as mental simulation , 2008, Trends in Cognitive Sciences.

[25]  H. Wood,et al.  In this issue , 2017, Nature Reviews Neuroscience.

[26]  N. Cowan What are the differences between long-term, short-term, and working memory? , 2008, Progress in brain research.

[27]  Jennifer Wiley,et al.  What Are the Odds? A Practical Guide to Computing and Reporting Bayes Factors , 2014, J. Probl. Solving.

[28]  Lysanne S. Post,et al.  Mechanisms of embodiment , 2015, Front. Psychol..

[29]  M. Hautus Corrections for extreme proportions and their biasing effects on estimated values ofd′ , 1995 .

[30]  G. Mazzoni,et al.  The role of kinematic mental simulation in creating false memories , 2018 .

[31]  R. Ellis,et al.  On the relations between seen objects and components of potential actions. , 1998, Journal of experimental psychology. Human perception and performance.

[32]  M. Jeannerod Neural Simulation of Action: A Unifying Mechanism for Motor Cognition , 2001, NeuroImage.

[33]  Monica Bucciarelli,et al.  How the construction of mental models improves learning , 2007 .

[34]  Angela R. Laird,et al.  ALE meta-analysis of action observation and imitation in the human brain , 2010, NeuroImage.

[35]  Rolf A. Zwaan,et al.  PSYCHOLOGICAL SCIENCE Research Article THE EFFECT OF IMPLIED ORIENTATION DERIVED FROM VERBAL CONTEXT ON PICTURE RECOGNITION , 2022 .

[36]  L. Barsalou Grounded cognition. , 2008, Annual review of psychology.

[37]  Rolf A. Zwaan,et al.  Seeing, acting, understanding: motor resonance in language comprehension. , 2006, Journal of experimental psychology. General.

[38]  G. Rizzolatti,et al.  Action recognition in the premotor cortex. , 1996, Brain : a journal of neurology.

[39]  P. Johnson-Laird,et al.  PSYCHOLOGICAL SCIENCE Research Article HOW DIAGRAMS CAN IMPROVE REASONING , 2022 .

[40]  A. Paivio,et al.  Picture superiority in free recall: Imagery or dual coding? , 1973 .

[41]  M. Erdelyi,et al.  Hypermnesia for pictures: Incremental memory for pictures but not words in multiple recall trials☆ , 1974 .

[42]  H. M. Müller,et al.  EEG beta-power changes reflect motor involvement in abstract action language processing , 2017, Brain and Language.

[43]  S. Gallagher Direct perception in the intersubjective context , 2008, Consciousness and Cognition.

[44]  Yann Coello,et al.  Perceiving what you intend to do from what you do: evidence for embodiment in social interactions , 2015, Socioaffective neuroscience & psychology.

[45]  P. Johnson-Laird,et al.  Mental models and temporal reasoning , 1996, Cognition.

[46]  D. McNeill,et al.  Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information , 1998 .

[47]  A. Glenberg,et al.  Processing Abstract Language Modulates Motor System Activity , 2008, Quarterly journal of experimental psychology.

[48]  Rolf A. Zwaan,et al.  Language Comprehenders Mentally Represent the Shapes of Objects , 2002, Psychological science.

[49]  Samuel T. Moulton,et al.  Imagining predictions: mental imagery as mental emulation , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[50]  Keith Stenning,et al.  Distinctions with Differences: Comparing Criteria for Distinguishing Diagrammatic from Sentential Systems , 2000, Diagrams.

[51]  Lawrence W. Barsalou,et al.  Language and simulation in conceptual processing , 2008 .

[52]  C. Urgesi,et al.  Simulating the future of actions in the human corticospinal system. , 2010, Cerebral cortex.

[53]  P. Johnson-Laird,et al.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness , 1985 .

[54]  Marcello Costantini,et al.  Grasping affordances with the other's hand: a TMS study. , 2013, Social cognitive and affective neuroscience.

[55]  M. Hegarty Mental animation: inferring motion from static displays of mechanical systems. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[56]  Randall W. Engle,et al.  Working Memory and Comprehension , 2004 .

[57]  Rolf A. Zwaan,et al.  Embodied Language: A Review of the Role of the Motor System in Language Comprehension , 2008, Quarterly journal of experimental psychology.

[58]  M. Candidi,et al.  Mapping Implied Body Actions in the Human Motor System , 2006, The Journal of Neuroscience.

[59]  U. Castiello The neuroscience of grasping , 2005, Nature Reviews Neuroscience.

[60]  Andrew A G Mattar,et al.  Motor Learning by Observing , 2005, Neuron.

[61]  J. Freyd,et al.  The mental representation of movement when static stimuli are viewed , 1983, Perception & psychophysics.

[62]  A. Baddeley Is working memory still working? , 2001, The American psychologist.

[63]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[64]  Leslie G. Ungerleider,et al.  Formation of a Motor Memory by Action Observation , 2005, The Journal of Neuroscience.