Analyzing polysemiosis: language, gesture, and depiction in two cultural practices with sand drawing

Abstract Human communication is by default polysemiotic: it involves the spontaneous combination of two or more semiotic systems, the most important ones being language, gesture, and depiction. We formulate an original cognitive-semiotic framework for the analysis of polysemiosis, contrasting this with more familiar systems based on the ambiguous term “multimodality.” To be fully explicit, we developed a coding system for the analysis of polysemiotic utterances containing speech, gesture, and drawing, and implemented this in the ELAN video annotation software. We used this to analyze 23 video-recordings of sand drawing performances on Paama, Vanuatu and 20 sand stories of the Pitjantjatjara culture in Central Australia. Methodologically we used the conceptual-empirical loop of cognitive semiotics: our theoretical framework guided general considerations, such as distinguishing between the “tiers” of gesture and depiction, and the three kinds of semiotic grounds (iconic, indexical, symbolic), but the precise decisions on how to operationalize these were made only after extensive work with the material. We describe the coding system in detail and provide illustrative examples from the Paamese and Pitjantjatjara data, remarking on both similarities and differences in the polysemiosis of the two cultural practices. We conclude by summarizing the contributions of the study and point to some directions for future research.

[1]  Simon Devylder.  The archipelago of meaning: Methodological contributions to the study of Vanuatu sand drawing , 2022, The Australian Journal of Anthropology.

[2]  David Rybák.  The issue of the ego in Husserl's Logical Investigations and Ideas Pertaining to a Pure Phenomenology and to a Phenomenological Philosophy. , 2020, E-LOGOS.

[3]  J. Zlatev,et al.  Pantomime as the original human-specific communicative system , 2020, Journal of Language Evolution.

[4]  J. Zlatev,et al.  Translating from monosemiotic to polysemiotic narratives: A study of Finnish speech and gestures , 2020 .

[5]  Georgios Stampoulidis.  Stories of resistance in Greek street Art: A cognitive-semiotic approach , 2019, Public Journal of Semiotics.

[6]  Simon Devylder.  Dancygier, Barbara (ed.), The Cambridge Handbook of Cognitive Linguistics , 2019, English Text Construction.

[7]  J. Zlatev.  Mimesis Theory, Learning, and Polysemiotic Communication , 2018, Encyclopedia of Educational Philosophy and Theory.

[8]  Simon Devylder.  Diagrammatic iconicity explains asymmetries in Paamese possessive constructions , 2018 .

[9]  Cornelia Müller.  Waking Metaphors: Embodied Cognition in Multimodal Discourse , 2017 .

[10]  Mutsumi Imai,et al.  The "symbol grounding problem" reinterpreted from the perspective of language acquisition , 2017 .

[11]  Sabine Tan,et al.  Intersemiotic Translation as Resemiotisation: A Multimodal Perspective , 2016 .

[12]  Jordan Zlatev,et al.  The emergence of gestures , 2015 .

[13]  J. Zlatev.  Human uniqueness, bodily mimesis and the evolution of language , 2014 .

[14]  Jennifer Green,et al.  Drawn from the Ground: Sound, Sign and Inscription in Central Australian Sand Stories , 2014 .

[15]  Mark Dingemanse,et al.  Advances in the Cross-Linguistic Study of Ideophones , 2012, Lang. Linguistics Compass.

[16]  G. Sonesson.  From mimicry to mime by way of mimesis: Reflections on a general theory of iconicity , 2010 .

[17]  Jordan Zlatev,et al.  Cross-modal iconicity: A cognitive semiotic approach to sound symbolism , 2010 .

[18]  Peter Hühn,et al.  Narration in Various Media , 2009 .

[19]  David Herman,et al.  The Cambridge Companion to Narrative , 2007 .

[20]  Marie-Laure Ryan,et al.  Toward a definition of narrative , 2007 .

[21]  H. Diessel.  Demonstratives, joint attention, and the emergence of grammar , 2006 .

[22]  Peter Wittenburg,et al.  ELAN: a Professional Framework for Multimodality Research , 2006, LREC.

[23]  M. Patterson.  Agency, Kinship, and History in North Ambrym , 2006 .

[24]  Knut M. Rio.  DISCUSSIONS AROUND A SAND‐DRAWING: CREATIONS OF AGENCY AND SOCIETY IN MELANESIA , 2005 .

[25]  John A. Taylor.  Paths of relationship, spirals of exchange: Imag(in)ing North Pentecost kinship , 2005 .

[26]  M. Arbib.  From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics , 2005, Behavioral and Brain Sciences.

[27]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[28]  F. Dussart.  Art and Agency: An Anthropological Theory. , 2000 .

[29]  S. Goldin-Meadow,et al.  Kanzi: The ape at the brink of the human mind , 1996, International Journal of Primatology.

[30]  Shinichi Tamura,et al.  Recognition of sign language motion images , 1988, Pattern Recognit..

[31]  N. Munn,et al.  Walbiri Iconography: Graphic Representation and Cultural Symbolism in a Central Australian Society , 1986 .

[32]  Eugenio Coseriu.  Linguistic Competence: What Is It Really? , 1985 .

[33]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[34]  R. Jakobson.  Quest for the Essence of Language , 1965 .

[35]  C. F. Hockett.  The origin of speech. , 1960, Scientific American.

[36]  C. K. Grant,et al.  On Using Language , 1956 .

[37]  F. D. Saussure.  Cours de linguistique générale , 1924 .

[38]  Ida Stevia Krogh Diget,et al.  Intersemiotic Translation from Film to Audio Description: A cognitive semiotic approach , 2019 .

[39]  Peter F. Wignell,et al.  Inter-semiotic Translation as Resemiotization: A Multimodal Perspective , 2017 .

[40]  B. Dancygier.  The Cambridge handbook of cognitive linguistics , 2017 .

[41]  C. Forceville.  Visual and Multimodal Metaphor in Advertising : Cultural Perspectives , 2017 .

[42]  Brian MacWhinney,et al.  The handbook of language emergence , 2015 .

[43]  P. Trifonas.  International handbook of semiotics , 2015 .

[44]  M. Engelmann.  The Philosophical Investigations , 2013 .

[45]  W. Sandler.  Dedicated gestures and the emergence of sign language , 2012 .

[46]  J. Zlatev,et al.  Cognitive Semiotics , 2016 .

[47]  J. Zlatev,et al.  Stages and transitions in children’s semiotic development , 2009 .

[48]  Göran Sonesson,et al.  From the meaning of embodiment to the embodiment of meaning , 2007 .

[49]  A. Umali,et al.  Hudhud and Noh : a dialogue of cultures : 2001 Unesco masterpieces of the oral and intangible heritage of humanity , 2007 .

[50]  Magnus Sahlgren,et al.  The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces , 2006 .

[51]  Vir V. Phoha,et al.  Viewpoint , 1999, CACM.

[52]  J. Zlatev.  Situated Embodiment : Studies in the emergence of spatial meaning , 1997 .

[53]  Göran Sonesson,et al.  Pictorial concepts. Inquiries into the semiotic heritage and its relevance for the analysis of the visual world , 1989 .

[54]  A. Deacon,et al.  Geometrical Drawings from Malekula and Other Islands of the New Hebrides. , 1934 .