Gesture and Language Production in Communication through Bar Graphs

Gesture and Language Production in Communication through Bar Graphs Ozge Alacam (alacam@informatik.uni-hamburg.de) Christopher Habel (habel@informatik.uni-hamburg.de) Cengiz Acarturk (acarturk@metu.edu.tr) Department of Informatics, University of Hamburg, Hamburg/Germany Cognitive Science, Middle East Technical University, Ankara/Turkey Abstract Bar graphs and line graphs are commonly used ways of graphical communication. Due to the difference in their perceptual visuo-spatial properties, they facilitate comprehension of different events. Bar graphs are commonly used in the domain of precipitation although the data intrinsically carry information that is averaged over long time spans. In this study, we investigate how the presence of incongruence between consecutive graph pairs influences conceptualization of the represented information about precipitation. For this, we analyzed gestures and verbal descriptions produced by the participants as indicators of event conceptualizations. The results of the experimental investigation reveals that when incongruent graph pairs are presented, the participants show tendency to produce directional gestures that accompany the verbal descriptions of the specific regions represented by one/two bars, indicating that bar graphs presented in consecutive order facilitates comprehension of trend information as well as of discrete entities. Additionally, the presence of incongruence seems to enhance the production of comparative words accompanied with non-directional gestures. Keywords: Gesture production; language production; bar graph comprehension; multimodal communication Visualization – Bar and Line Graphs The primary goal of visualizing data is to (re-)present them in a format more suitable for using them in thinking, problem solving and communication (This view is taken implicitly or explicitly in most seminal publications on graphs, as well as on visualization during the last decades, see, e.g., Tufte 1983, Kosslyn 1989, 2006, Hegarty 2011). Line graphs and bar graphs are successful means to present data, both in the task of analyzing the data and in the task of communicating the results of data analysis. Communicating visualized data using bars or lines is used extensively in scientific publications, textbooks, magazines and news- papers; Zacks, Levy, Tversky, & Schiano’s (2002) study on the use of graphs in the print media shows that line graphs and bar graphs are the dominant, i.e. most frequently used, types of graphs in addressing non-experts in communication through graphs. The primary gain in using graphs is not to make individual data points visible but to provide visual access to relations between data points (‘x1-y1 has a larger y-value than x2-y2’) or to second-order entities as ‘trends’. This advantage can be ascribed to humans’ pattern perception processes, in particular visual chunking (see, Shah, Mayer & Hegarty, 1999). Beyond these commonalities, there seem to be functional differences between bar graphs and line graphs. Zacks and Tversky (1999) investigate the bar–line message correspondence, which considers the systematic relations between the type of graph used and the type of message intended to be communicated. Zacks and Tversky point to a preferred “use of bar graphs to depict comparisons among discrete data points, and line graphs to depict trends” (p. 1073). On the other hand, participants in their experiments had a strong tendency for relational descriptions (e.g., “A is higher than B”) after comprehending bar graphs and for process-oriented second- order descriptions as ‘trends’ (e.g., “X increases from A to B”) in the of line graphs (p. 1078). Shah, Meyer and Hegarty (1999) report—with respect to these tendencies—a comparable view, but in presenting their perceptual organization hypothesis they lay an additional focus on Gestalt principles realizable in the graph types in question. In addition to text-graphics documents, in many professional communication settings as well as in classroom settings, graphs, spoken language, and often gestures, accompany each other forming multimodal communication. In dynamic communication of this type, often recipients have to integrate messages communicated by a sequence of graphs. The present study investigates participants’ verbal descriptions of pairs of succeeding bar graphs and the gestures produced during these descriptions. The first graph of each pair depicts averages (monthly precipitation over three decades) whereas the second graph depicts instances (monthly precipitation of a specific year). Due to the average-instance constellation, commonalities and differences, which we regard as ‘incongruences’, between the graphs play a major role in comprehending the graphs and in following production of verbal descriptions; in this setting the within-the-bar bias (Newman & Scholl, 2012) did not occur. Gesture and Language The studies on gesture-language interaction are mainly based on the assumption that concepts are sensorimotor, by emphasizing that they are grounded in physical world and based on perceptual experience (Barsalou, 1999; Garbarini & Adenzato, 2004). There are several frameworks that investigate gestures from various perspectives, but all of them agree on that gestures rely on spatial representations. According to the GSA framework (‘Gesture-as-simulated-