Sound Predicts Meaning: Cross-Modal Associations Between Formant Frequency and Emotional Tone in Stanzas

Research on the relation between sound and meaning in language has reported substantial evidence for implicit associations between articulatory-acoustic characteristics of phonemes and emotions. In the present study, we specifically tested the relation between the acoustic properties of a text and its emotional tone as perceived by readers. To this end, we asked participants to assess the emotional tone of single stanzas extracted from a large variety of poems. The selected stanzas had either an extremely high, a neutral, or an extremely low average formant dispersion. To assess the average formant dispersion per stanza, all words were phonetically transcribed and the distance between the first and second formant per vowel was calculated. Building on a long tradition of research on associations between sound frequency on the one hand and non-acoustic concepts such as size, strength, or happiness on the other hand, we hypothesized that stanzas with an extremely high average formant dispersion would be rated lower on items referring to Potency (dominance) and higher on items referring to Activity (arousal) and Evaluation (emotional valence). The results confirmed our hypotheses for the dimensions of Potency and Evaluation, but not for the dimension of Activity. We conclude that, at least in poetic language, extreme values of acoustic features of vowels are a significant predictor for the emotional tone of a text.

[1]  Jan Auracher,et al.  P Is for Happiness, N Is for Sadness: Universals in Sound Iconicity to Detect Emotions in Poetry , 2010 .

[2]  David M. Sidhu,et al.  Five mechanisms of sound symbolic association , 2017, Psychonomic Bulletin & Review.

[3]  A. Simpson,et al.  Acoustic analysis of German vowels in the Kiel Corpus of Read Speech , 1997 .

[4]  Stanley S. Newman.  Further experiments in phonetic symbolism , 1933 .

[5]  Marlene Johansson Falck,et al.  Iconic Prosody in Story Reading , 2015, Cogn. Sci..

[6]  Maria Kraxenberger,et al.  Mimological Reveries? Disconfirming the Hypothesis of Phono-Emotional Iconicity in Poetry , 2016, Front. Psychol..

[7]  Magdalena Wrembel,et al.  "sounds like a Rainbow" - Sound-colour Mappings in Vowel Perception , 2008, ExLing.

[8]  Edward Gibson,et al.  Wordform Similarity Increases With Semantic Similarity: An Analysis of 100 Languages , 2016, Cogn. Sci..

[9]  Martine Grice,et al.  Mood is linked to vowel type: the role of articulatory movements. , 2014, Emotion.

[10]  L. Elizabeth Crawford,et al.  Conceptual Metaphors of Affect , 2009 .

[11]  W. Fitch.  Vocal tract length and formant frequency dispersion correlate with body size in rhesus macaques. , 1997, The Journal of the Acoustical Society of America.

[12]  Richard R. Klink.  Creating Brand Names With Meaning: The Use of Sound Symbolism , 2000 .

[13]  Marc O Ernst,et al.  Natural auditory scene statistics shapes human spatial hearing , 2014, Proceedings of the National Academy of Sciences.

[14]  Michal Kouril,et al.  Sounds good: Phonetic sound patterns in top brand names , 2014, Marketing Letters.

[15]  G. Lupyan,et al.  Iconicity in English and Spanish and Its Relation to Lexical Category and Age of Acquisition , 2015, PloS one.

[16]  C. Spence.  Crossmodal correspondences: A tutorial review , 2011, Attention, perception & psychophysics.

[17]  Martin Kuška,et al.  Modeling Semantic Emotion Space Using a 3D Hypercube-Projection: An Innovative Analytical Approach for the Psychology of Emotions , 2016, Front. Psychol..

[18]  David S. Miall.  Sounds of contrast: an empirical approach to phonemic iconicity , 2001 .

[19]  C. Whissell,et al.  Phonosymbolism and the Emotional Nature of Sounds: Evidence of the Preferential Use of Particular Phonemes in Texts of Differing Emotional Tone , 1999, Perceptual and motor skills.

[20]  Siddharth Dalmia,et al.  Epitran: Precision G2P for Many Languages , 2018, LREC.

[21]  A. Jacobs,et al.  Extracting salient sublexical units from written texts: “Emophon,” a corpus-based approach to phonological iconicity , 2013, Front. Psychol..

[22]  I. Fónagy.  Communication in Poetry , 1961 .

[23]  Simon Kirby,et al.  Cross-modal associations and synesthesia: Categorical perception and structure in vowel–color mappings in a large online sample , 2019, Behavior Research Methods.

[24]  Cynthia Whissell,et al.  The Emotional Symbolism of Two English E-Sounds: /I/ as in “Cheap” is Pleasant and /I/ as in “Chip” Active , 2003, Perceptual and motor skills.

[25]  J. Jenkins,et al.  Studies in the Psychological Correlates of the Sound System of American English , 1964 .

[26]  S. Kita,et al.  Sound symbolism facilitates early verb learning , 2008, Cognition.

[27]  B. Rossion,et al.  Fixation Patterns During Recognition of Personally Familiar and Unfamiliar Faces , 2010, Front. Psychology.

[28]  J. Russell,et al.  An approach to environmental psychology , 1974 .

[29]  Reuven Tsur,et al.  Methodological issues in the study of phonetic symbolism , 2019 .

[30]  D. Massaro,et al.  Iconicity in the speech of children and adults. , 2018, Developmental science.

[31]  D. Lüdecke.  Sjplot - Data Visualization For Statistics In Social Science. , 2018 .

[32]  L. Nygaard,et al.  The Semantics of Prosody: Acoustic and Perceptual Evidence of Prosodic Correlates to Word Meaning , 2009, Cogn. Sci..

[33]  Mark Dingemanse,et al.  Iconicity in the lab: a review of behavioral, developmental, and neuroimaging research into sound-symbolism , 2015, Front. Psychol..

[34]  Richard R. Klink,et al.  The role of position, type, and combination of sound symbolism imbeds in brand names , 2014 .

[35]  Hanako Yoshida,et al.  A Cross-Linguistic Study of Sound Symbolism in Children's Verb Learning , 2012, Journal of cognition and development : official journal of the Cognitive Development Society.

[36]  P. Hagoort,et al.  Sound-symbolism boosts novel word learning. , 2016, Journal of experimental psychology. Learning, memory, and cognition.

[37]  A. Jacobs,et al.  Phonological iconicity , 2014, Front. Psychol..

[38]  J. Ohala.  Sound symbolism: The frequency code underlies the sound-symbolic use of voice pitch , 1995 .

[39]  L. Nygaard,et al.  Sound to meaning correspondences facilitate word learning , 2009, Cognition.

[40]  L. Marks.  The Unity of the Senses: Interrelations Among the Modalities , 1978 .

[41]  Tina M. Lowrey,et al.  Sound symbolism effects across languages: Implications for global brand names , 2012 .

[42]  秋田 喜美.  A grammar of sound-symbolic words in Japanese : theoretical approaches to iconic and lexical properties of mimetics , 2009 .

[43]  P. Gilbert,et al.  New theoretical conceptions of social anxiety and social phobia , 1989 .

[44]  Mutsumi Imai,et al.  Japanese Sound-Symbolism Facilitates Word Learning in English-Speaking Children , 2011, Cogn. Sci..

[45]  Jan Auracher.  Sound iconicity of abstract concepts: Place of articulation is implicitly associated with abstract concepts of size and social dominance , 2017, PloS one.

[46]  G. Leibniz,et al.  Nouveaux essais sur l'entendement humain / , 1922 .

[47]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[48]  Galia Angelova,et al.  Recent advances in natural language processing , 1997 .

[49]  D. Reby,et al.  The evolution of acoustic size exaggeration in terrestrial mammals , 2016, Nature Communications.

[50]  J. Nuckolls.  THE CASE FOR SOUND SYMBOLISM , 1999 .

[51]  Nick Neave,et al.  Relationships between vocal characteristics and body size and shape in human males: An evolutionary explanation for a deep male voice , 2006, Biological Psychology.

[52]  E. Fischer‐Jorgensen,et al.  Perceptual Dimensions of Vowels , 1968 .

[53]  P. Laukka,et al.  A dimensional approach to vocal expression of emotion , 2005 .

[54]  S. Gaulin,et al.  Men's voices as dominance signals: vocal fundamental and formant frequencies influence dominance attributions among men , 2007 .

[55]  Charles E. Osgood,et al.  A Cross-Cultural Study of the Affective Meanings of Color , 1973 .

[56]  Klaus R. Scherer,et al.  Emotion dimensions and formant position , 2009, INTERSPEECH.

[57]  J. M. Kittross.  The measurement of meaning , 1959 .

[58]  Mathias Scharinger,et al.  Contiguity-based sound iconicity: The meaning of words resonates with phonetic properties of their immediate verbal contexts , 2019, PloS one.

[59]  Laura L Namy,et al.  Say It Like You Mean It: Mothers’ Use of Prosody to Convey Word Meaning , 2012, Language and speech.

[60]  M. S. Miron,et al.  A cross-linguistic investigation of phonetic symbolism. , 1961, Journal of abnormal and social psychology.

[61]  David R. Simmons,et al.  Cross-modal associations in synaesthesia: Vowel colours in the ear of the beholder , 2014, i-Perception.

[62]  P. Monaghan,et al.  Do sound symbolism effects for written words relate to individual phonemes or to phoneme features? , 2019, Language and Cognition.

[63]  S K Fisher,et al.  Comparison of associations to vowel speech sounds by English and Spanish speakers. , 1988, The American journal of psychology.

[64]  Katarzyna Pisanski,et al.  Vocal indicators of dominance , 2016 .

[65]  Max Louwerse,et al.  Estimating valence from the sound of a word: Computational, experimental, and cross-linguistic evidence , 2017, Psychonomic bulletin & review.

[66]  Reuven Tsur,et al.  Size–sound symbolism revisited , 2006 .

[67]  Geir Kjetsaa.  Lomonosov's sound characteristics , 1974 .

[68]  Gabriella Vigliocco,et al.  Iconicity as a General Property of Language: Evidence from Spoken and Signed Languages , 2010, Front. Psychology.

[69]  R. G. Kent,et al.  Language: Its Nature, Development, and Origin , 1923 .

[70]  Jan Auracher,et al.  Commentary on Tsur and Gafni , 2019 .

[71]  Etsuko Haryu,et al.  Investigation of the process underpinning vowel‐size correspondence. , 2013 .

[72]  A. Jacobs,et al.  The Sound of Words Evokes Affective Brain Responses , 2018, Brain sciences.

[73]  Cynthia Whissell,et al.  Emotional Sound Symbolism and the Volta in Shakespearean and Petrarchan Sonnets , 2018 .

[74]  Charles P. Davis,et al.  What Does a Horgous Look Like? Nonsense Words Elicit Meaningful Drawings , 2019, Cogn. Sci..

[75]  Linda R. Waugh,et al.  The Sound Shape of Language , 1979 .

[76]  J. Ohala.  Sound Symbolism , 2004, Encyclopedia of Slavic Languages and Linguistics Online.

[77]  Cynthia Whissell.  “The Sound Must Seem an Echo to the Sense”: Pope's Use of Sound to Convey Meaning in His Translation of Homer's Iliad , 2004, Perceptual and motor skills.

[78]  Eric Yorkston,et al.  A Sound Idea: Phonetic Effects of Brand Names on Consumer Judgments , 2004 .

[79]  J. Adelman,et al.  Emotional sound symbolism: Languages rapidly signal valence via phonemes , 2018, Cognition.

[80]  Michael D. Robinson,et al.  When "light" and "dark" thoughts become light and dark responses: affect biases brightness judgments. , 2007, Emotion.

[81]  Tina M. Lowrey,et al.  Phonetic Symbolism and Brand Name Preference , 2007 .

[82]  Morten H. Christiansen,et al.  Arbitrariness, Iconicity, and Systematicity in Language , 2015, Trends in Cognitive Sciences.

[83]  Cynthia Whissell.  Sound and Emotion in Milton's Paradise Lost , 2011, Perceptual and motor skills.

[84]  Ivan Fonagy.  Die Metaphern in der Phonetik : ein Beitrag zur Entwicklungsgeschichte des wissenschaftlichen Denkens , 1963 .

[85]  Vivi Nastase,et al.  Do Happy Words Sound Happy ? A study of the relation between form and meaning for English words expressing emotions , .

[86]  Madison Bentley,et al.  An accessory study of “phonetic symbolism” , 1933 .

[87]  Cesare Parise,et al.  Audiovisual Cross-Modal Correspondences in the General Population , 2013 .

[88]  Robert D. Tarte,et al.  The relationship between monosyllables and pure tones: An investigation of phonetic symbolism , 1982 .

[89]  Douglas D. O'Shaughnessy,et al.  Speech communications - human and machine, 2nd Edition , 2000 .

[90]  Eli D. Strauss,et al.  Aggression and dominance: an interdisciplinary overview , 2016, Current Opinion in Behavioral Sciences.

[91]  Morten H. Christiansen,et al.  Sound–meaning association biases evidenced across thousands of languages , 2016, Proceedings of the National Academy of Sciences.

[92]  Alina Catalina Duduciuc.  ADVERTISING BRANDS BY MEANS OF SOUNDS SYMBOLISM: THE INFLUENCE OF VOWELS ON PERCEIVED BRAND CHARACTERISTICS , 2015 .

[93]  A. Jacobs,et al.  Measuring the Basic Affective Tone of Poems via Phonological Saliency and Iconicity , 2016 .

[94]  D. R. Heise.  Sound-Meaning Correlations Among 1,000 English Words , 1966 .

[95]  Ingo Siegert,et al.  Vowels formants analysis allows straightforward detection of high arousal emotions , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[96]  Keith S. Coulter.  The Effects of Phonetic Symbolism on Comparative Price Perceptions , 2009 .

[97]  Cynthia Whissell.  Sound Symbolism in Shakespeare’s Sonnets: Evidence of Dramatic Tension in the Interplay of Harsh and Gentle Sounds , 2017 .

[98]  Terry K Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal Chiropractic Medicine.

[99]  Nahyun Kwon,et al.  Semantic Associations Dominate Over Perceptual Associations in Vowel–Size Iconicity , 2019, i-Perception.

[100]  Keith S. Coulter,et al.  Small Sounds, Big Deals: Phonetic Symbolism Effects in Pricing , 2010 .

[101]  Simon Kirby,et al.  How arbitrary is language? , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[102]  S. Kita,et al.  The sound symbolism bootstrapping hypothesis for language acquisition and language evolution , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[103]  E. Morton.  On the Occurrence and Significance of Motivation-Structural Rules in Some Bird and Mammal Sounds , 1977, The American Naturalist.

[104]  P. Vink,et al.  Pleasure, Arousal, Dominance: Mehrabian and Russell revisited , 2014, Current Psychology.

[105]  S Ertel.  [The sound character of artificial sound structures]. , 1965, Psychologische Forschung.

[106]  Cynthia Whissell,et al.  Poetic Emotion and Poetic Style: The 100 Poems Most Frequently Included in Anthologies and the Work of Emily Dickinson , 2004 .

[107]  Geoff Hollis,et al.  Weighing up the evidence for sound symbolism: Distributional properties predict cue strength , 2017 .

[108]  M. Koenigs,et al.  Inherent emotional quality of human speech sounds , 2013, Cognition & emotion.

[109]  Cynthia Whissell,et al.  An Episodic Plot and an Unhappy Ending , 2016, Psychological reports.

[110]  Carlyle Folkins,et al.  An Investigation of the Expressive Values of Graphemes , 1966 .

[111]  A. Jacobs,et al.  Why 'piss' is ruder than 'pee'? The role of sound in affective meaning making , 2018, PloS one.

[112]  E. Sapir.  A study in phonetic symbolism. , 1929 .

[113]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[114]  David Philippou-Hübner,et al.  Vowels Formants Analysis Allows Straightforward Detection of High Arousal Acted and Spontaneous Emotions , 2011, INTERSPEECH.

[115]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[116]  W. Fitch,et al.  Morphology and development of the human vocal tract: a study using magnetic resonance imaging. , 1999, The Journal of the Acoustical Society of America.