Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inference

Understanding how language supports emotion inference remains a topic of debate in emotion science. The present study investigated whether language-derived emotion-concept knowledge would causally support emotion inference by manipulating the language-specific knowledge representations in large language models. Using the prompt technique, 14 attributes of emotion concepts were found to be represented by distinct artificial neuron populations. By manipulating these attribute-related neurons, the majority of the emotion inference tasks showed performance deterioration compared to random manipulations. The attribute-specific performance deterioration was related to the importance of different attributes in human mental space. Our findings provide causal evidence in support of a language-based mechanism for emotion inference and highlight the contributions of emotion-concept knowledge.

[1]  D. Barch,et al.  Large-scale encoding of emotion concepts becomes increasingly similar between individuals from childhood to adolescence , 2023, Nature Neuroscience.

[2]  E. Viding,et al.  Genetic algorithms reveal profound individual differences in emotion recognition , 2022, Proceedings of the National Academy of Sciences of the United States of America.

[3]  Alexander G. Huth,et al.  Semantic reconstruction of continuous language from non-invasive brain recordings , 2022, bioRxiv.

[4]  Ajay B. Satpute,et al.  The cultural evolution of emotion , 2022, Nature Reviews Psychology.

[5]  Evelina Fedorenko,et al.  Semantic projection recovers rich human knowledge of multiple object features from word embeddings , 2022, Nature Human Behaviour.

[6]  A. Hyafil,et al.  The cultural evolution of love in literary history , 2022, Nature Human Behaviour.

[7]  Omer Levy,et al.  Shared computational principles for language processing in humans and deep language models , 2022, Nature Neuroscience.

[8]  J. King,et al.  Brains and algorithms partially converge in natural language processing , 2022, Communications Biology.

[9]  Xin Hu,et al.  Similar brains blend emotion in similar ways: Neural representations of individual difference in emotion profiles , 2021, NeuroImage.

[10]  Alexander G. Huth,et al.  Visual and linguistic semantic representations are aligned at the border of human visual cortex , 2021, Nature Neuroscience.

[11]  Thomas P. Reber,et al.  Concept neurons in the human medial temporal lobe flexibly represent abstract relations between concepts , 2021, Nature Communications.

[12]  Y. Bi Dual coding of knowledge in the human brain , 2021, Trends in Cognitive Sciences.

[13]  Hiroaki Hayashi,et al.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing , 2021, ACM Comput. Surv..

[14]  Marcelo Mendoza,et al.  Inspecting the concept knowledge graph encoded by modern language models , 2021, FINDINGS.

[15]  Liqin Zhou,et al.  Emerged human-like facial expression representation in a deep convolutional neural network , 2021, bioRxiv.

[16]  Henrik Saalbach,et al.  Let’s Talk About Emotions: the Development of Children’s Emotion Vocabulary from 4 to 11 Years of Age , 2021, Affective Science.

[17]  Rajeev D. S. Raizada,et al.  Deep Artificial Neural Networks Reveal a Distributed Cortical Network Encoding Propositional Sentence-Level Meaning , 2021, The Journal of Neuroscience.

[18]  Kristen A. Lindquist,et al.  At the Neural Intersection Between Language and Emotion , 2021, Affective Science.

[19]  C. Rothkopf,et al.  Large pre-trained language models contain human-like biases of what is right and wrong to do , 2021, Nature Machine Intelligence.

[20]  Omer Levy,et al.  Transformer Feed-Forward Layers Are Key-Value Memories , 2020, EMNLP.

[21]  Alan S. Cowen,et al.  Semantic Space Theory: A Computational Approach to Emotion , 2020, Trends in Cognitive Sciences.

[22]  V. Kuperman,et al.  How emotion is learned: Semantic learning of novel words in emotional contexts , 2020, Journal of Memory and Language.

[23]  L. F. Barrett,et al.  The N400 indexes acquisition of novel emotion concepts via conceptual combination. , 2020, Psychophysiology.

[24]  B. Repacholi,et al.  Beyond Language in Infant Emotion Concept Development , 2020 .

[25]  B. Repacholi,et al.  Do Preverbal Infants Understand Discrete Facial Expressions of Emotion? , 2020, Emotion Review.

[26]  Desmond C. Ong,et al.  The contribution of linguistic and visual cues to physiological synchrony and empathic accuracy , 2020, Cortex.

[27]  Eghbal A. Hosseini,et al.  The neural architecture of language: Integrative modeling converges on predictive processing , 2020, Proceedings of the National Academy of Sciences.

[28]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[29]  Emily Denton,et al.  Social Biases in NLP Models as Barriers for Persons with Disabilities , 2020, ACL.

[30]  Anna Rumshisky,et al.  A Primer in BERTology: What We Know About How BERT Works , 2020, Transactions of the Association for Computational Linguistics.

[31]  Kristen A. Lindquist,et al.  Comment: A role of Language in Infant Emotion Concept Acquisition , 2020 .

[32]  L. F. Barrett,et al.  Comment: Emotions Are Abstract, Conceptual Categories That Are Learned by a Predicting Brain , 2020 .

[33]  Henrik Saalbach,et al.  Emotion-specific vocabulary and its contribution to emotion understanding in 4- to 9-year-old children. , 2020, Journal of experimental child psychology.

[34]  Kristen A. Lindquist,et al.  How do children learn novel emotion words? A study of emotion concept acquisition in preschoolers. , 2019, Journal of experimental psychology. General.

[35]  Alan S. Cowen,et al.  The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain Regions , 2019, bioRxiv.

[36]  L. F. Barrett,et al.  Developing an Understanding of Emotion Categories: Lessons from Objects , 2019, Trends in Cognitive Sciences.

[37]  Ajay B. Satpute,et al.  The Default Mode Network’s Role in Discrete Emotion , 2019, Trends in Cognitive Sciences.

[38]  Marco Marelli,et al.  Vector-Space Models of Semantic Representation From a Cognitive Perspective: A Discussion of Common Misconceptions , 2019, Perspectives on psychological science : a journal of the Association for Psychological Science.

[39]  Fei Xu,et al.  Emotion words, emotion concepts, and emotional development in children: A constructionist hypothesis. , 2019, Developmental psychology.

[40]  R. Samuels,et al.  Could emotion development really be the acquisition of emotion concepts? , 2019, Developmental psychology.

[41]  Allyson Ettinger,et al.  What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models , 2019, TACL.

[42]  N. Sadato,et al.  The neural representation of facial-emotion categories reflects conceptual structure , 2019, Proceedings of the National Academy of Sciences.

[43]  Aleix M. Martinez,et al.  Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements , 2019, Psychological science in the public interest : a journal of the American Psychological Society.

[44]  Alex Wang,et al.  What do you learn from context? Probing for sentence structure in contextualized word representations , 2019, ICLR.

[45]  Noe Casas,et al.  Evaluating the Underlying Gender Bias in Contextualized Word Embeddings , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[46]  D. Whitney,et al.  Tracking the affective state of unseen persons , 2019, Proceedings of the National Academy of Sciences.

[47]  Alan S. Cowen,et al.  What Basic Emotion Theory Really Says for the Twenty-First Century Study of Emotion , 2019, Journal of Nonverbal Behavior.

[48]  Jeffrey A. Brooks,et al.  Conceptual knowledge predicts the representational structure of facial emotion perception , 2018, Nature Human Behaviour.

[49]  R. Adolphs,et al.  Investigating Emotions as Functional States Distinct From Feelings , 2018, Emotion review : journal of the International Society for Research on Emotion.

[50]  Johnny R. J. Fontaine,et al.  The semantic structure of emotion words across languages is consistent with componential appraisal models of emotion , 2018, Cognition & emotion.

[51]  Nancy Kanwisher,et al.  Toward a universal decoder of linguistic meaning from brain activation , 2018, Nature Communications.

[52]  M. Nitsche,et al.  Studying and modifying brain function with non-invasive brain stimulation , 2018, Nature Neuroscience.

[53]  Kristen A. Lindquist The role of language in emotion: existing evidence and future directions. , 2017, Current opinion in psychology.

[54]  Robert P. Spunt,et al.  The neuroscience of understanding the emotions of others , 2017, Neuroscience Letters.

[55]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[56]  L. F. Barrett The theory of constructed emotion: an active inference account of interoception and categorization , 2017, Social cognitive and affective neuroscience.

[57]  Erik C. Nook,et al.  Increasing verbal knowledge mediates development of multidimensional emotion representations , 2017, Nature Human Behaviour.

[58]  R. Adolphs,et al.  Emotion Perception from Face, Voice, and Touch: Comparisons and Convergence , 2017, Trends in Cognitive Sciences.

[59]  Nikola Vukovic,et al.  Primary motor cortex functionally contributes to language comprehension: An online rTMS study , 2017, Neuropsychologia.

[60]  F. Binkofski,et al.  The challenge of abstract concepts. , 2017, Psychological bulletin.

[61]  Kristen A. Lindquist,et al.  Bodily Contributions to Emotion: Schachter’s Legacy for a Psychological Constructionist View on Emotion , 2017 .

[62]  François-Laurent De Winter,et al.  Functional dissociation between anterior temporal lobe and inferior frontal gyrus in the processing of dynamic body expressions: Insights from behavioral variant frontotemporal dementia , 2016, Human brain mapping.

[63]  J. Binder In defense of abstract conceptual representations , 2016, Psychonomic Bulletin & Review.

[64]  T. Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal Chiropractic Medicine.

[65]  Geoff Hollis,et al.  The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics , 2016, Psychonomic Bulletin & Review.

[66]  Alexander G. Huth,et al.  Natural speech reveals the semantic maps that tile human cerebral cortex , 2016, Nature.

[67]  R. Saxe,et al.  Neural Representations of Emotion Are Organized around Abstract Event Features , 2015, Current Biology.

[68]  R. Fitzpatrick,et al.  Floor and ceiling effects in the OHS: an analysis of the NHS PROMs data set , 2015, BMJ Open.

[69]  Erik C. Nook,et al.  A new look at emotion perception: Concepts speed and shape facial emotion recognition. , 2015, Emotion.

[70]  Seong-Whan Lee,et al.  Abstract Representations of Associated Emotions in the Human Brain , 2015, The Journal of Neuroscience.

[71]  Ajay B. Satpute,et al.  Does Language Do More Than Communicate Emotion? , 2015, Current directions in psychological science.

[72]  Rebecca Saxe,et al.  A Common Neural Code for Perceived and Inferred Emotion , 2014, The Journal of Neuroscience.

[73]  D. Klass,et al.  A Social Constructionist Account of Grief: Loss and the Narration of Meaning , 2014, Death studies.

[74]  Yong Tao,et al.  Compound facial expressions of emotion , 2014, Proceedings of the National Academy of Sciences.

[75]  Lisa Feldman Barrett,et al.  Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. , 2014, Emotion.

[76]  Gert Westermann,et al.  From perceptual to language-mediated categorization , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[77]  Kristen A. Lindquist Emotions Emerge from More Basic Psychological Ingredients: A Modern Psychological Constructionist Model , 2013 .

[78]  A. Ortony,et al.  Psychological Construction in the OCC Model of Emotion , 2013, Emotion review : journal of the International Society for Research on Emotion.

[79]  F. Pulvermüller How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics , 2013, Trends in Cognitive Sciences.

[80]  Anqi Qiu,et al.  Emotional expressions in voice and music: Same code, same effect? , 2013, Human brain mapping.

[81]  Antonio Rangel,et al.  Stimulus Value Signals in Ventromedial PFC Reflect the Integration of Attribute Value Signals Computed in Fusiform Gyrus and Posterior Superior Temporal Gyrus , 2013, The Journal of Neuroscience.

[82]  Rick Dale,et al.  Assessing bimodality to detect the presence of a dual cognitive process , 2013, Behavior research methods.

[83]  Kristen A. Lindquist,et al.  The brain basis of emotion: A meta-analytic review , 2012, Behavioral and Brain Sciences.

[84]  Andrea Scarantino,et al.  Functional specialization does not require a one-to-one mapping between brain regions and emotions , 2012, Behavioral and Brain Sciences.

[85]  Kristen A. Lindquist,et al.  What are emotions and how are they created in the brain? , 2012, Behavioral and Brain Sciences.

[86]  P. Ekman,et al.  What is Meant by Calling Emotions Basic , 2011 .

[87]  Jessica L. Tracy,et al.  Four Models of Basic Emotions: A Review of Ekman and Cordaro, Izard, Levenson, and Panksepp and Watt , 2011 .

[88]  R. Levenson Basic Emotion Questions , 2011 .

[89]  G. Vigliocco,et al.  The representation of abstract words: why emotion matters. , 2011, Journal of experimental psychology. General.

[90]  Erez Lieberman Aiden,et al.  Quantitative Analysis of Culture Using Millions of Digitized Books , 2010, Science.

[91]  Vladimir M. Sloutsky,et al.  From Perceptual Categories to Concepts: What Develops? , 2010, Cogn. Sci..

[92]  Anthony D. Wagner,et al.  Neural Priming in Human Frontal Cortex: Multiple Forms of Learning Reduce Demands on the Prefrontal Executive System , 2009, Journal of Cognitive Neuroscience.

[93]  Jamil Zaki,et al.  Unpacking the informational bases of empathic accuracy. , 2009, Emotion.

[94]  Bradford Z. Mahon,et al.  A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content , 2008, Journal of Physiology-Paris.

[95]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[96]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[97]  J. Hodges,et al.  Semantic dementia: a unique clinicopathological syndrome , 2007, The Lancet Neurology.

[98]  M. Hallett Transcranial Magnetic Stimulation: A Primer , 2007, Neuron.

[99]  Jeffrey T. Hancock,et al.  Expressing emotion in text-based communication , 2007, CHI.

[100]  L. F. Barrett Are Emotions Natural Kinds? , 2006, Perspectives on psychological science : a journal of the Association for Psychological Science.

[101]  L. F. Barrett Solving the Emotion Paradox: Categorization and the Experience of Emotion , 2006, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[102]  Kristen A. Lindquist,et al.  Language and the perception of emotion. , 2006, Emotion.

[103]  S. Kotz,et al.  Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing , 2006, Trends in Cognitive Sciences.

[104]  A. Young,et al.  Priming of Emotion Recognition , 2005, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[105]  Daniel L. Schacter,et al.  Specificity of priming: a cognitive neuroscience perspective , 2004, Nature Reviews Neuroscience.

[106]  M. Cabanac What is emotion? , 2002, Behavioural Processes.

[107]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[108]  Alan Cowey,et al.  Transcranial magnetic stimulation and cognitive neuroscience , 2000, Nature Reviews Neuroscience.

[109]  Lisa Maxfield,et al.  Attention and Semantic Priming: A Review of Prime Task Effects , 1997, Consciousness and Cognition.

[110]  P. Ekman An argument for basic emotions , 1992 .

[111]  J. Dunn,et al.  Family Talk about Feeling States and Children's Later Understanding of Others' Emotions. , 1991 .

[112]  Ira J. Roseman,et al.  Appraisals of emotion-eliciting events: Testing a theory of discrete emotions. , 1990 .

[113]  N. C. Silver,et al.  Averaging Correlation Coefficients: Should Fishers z Transformation Be Used? , 1987 .

[114]  J. Hartigan,et al.  The Dip Test of Unimodality , 1985 .

[115]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[116]  Allyson Ettinger,et al.  COMPS: Conceptual Minimal Pair Sentences for testing Property Knowledge and Inheritance in Pre-trained Language Models , 2022, ArXiv.

[117]  David Sander,et al.  Theoretical approaches to emotion and its measurement , 2021, Emotion Measurement.

[118]  Jungo Kasai,et al.  Understanding Commonsense Inference Aptitude of Deep Contextual Representations , 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing.

[119]  Sudeep Bhatia Associative Judgment and Vector Space Semantics , 2017, Psychological review.

[120]  Alan J. Fridlund,et al.  The behavioral ecology view of facial displays, 25 years later , 2017 .

[121]  Kristen A. Lindquist,et al.  Emotion perception, but not affect perception, is impaired with semantic memory loss. , 2014, Emotion.

[122]  J. Russell Core affect and the psychological construction of emotion. , 2003, Psychological review.

[123]  D. R. Lehman,et al.  Resilience to loss and chronic grief: a prospective study from preloss to 18-months postloss. , 2002, Journal of personality and social psychology.

[124]  Journal Pre-proof Transcranial direct current stimulation of the right anterior temporal lobe changes interpersonal neural synchronization and shared mental processes , 2022 .

[125]  Lisa Feldman Barrett,et al.  Emotion Words Shape Emotion Percepts , 2022 .

[126]  Chaz Firestone,et al.  " Top-down " Effects Where None Should Be Found: the El Greco Fallacy in Perception Research , 2022 .