Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts

Charts often contain visually prominent features that draw attention to aspects of the data and include text captions that emphasize aspects of the data. Through a crowdsourced study, we explore how readers gather takeaways when considering charts and captions together. We first ask participants to mark visually prominent regions in a set of line charts. We then generate text captions based on the prominent features and ask participants to report their takeaways after observing chart-caption pairs. We find that when both the chart and caption describe a high-prominence feature, readers treat the doubly emphasized high-prominence feature as the takeaway; when the caption describes a low-prominence chart feature, readers rely on the chart and report a higher-prominence feature as the takeaway. We also find that external information that provides context, helps further convey the caption’s message to the reader. We use these findings to provide guidelines for authoring effective chart-caption pairs.

[1]  Niklas Elmqvist,et al.  DataSite: Proactive visual data exploration with computation of insight-based recommendations , 2018, Inf. Vis..

[2]  Johanna D. Moore,et al.  Generating Explanatory Captions for Information Graphics , 1995, IJCAI.

[3]  Claire Lauer,et al.  Testing the Susceptibility of Users to Deceptive Data Visualizations When Paired with Explanatory Text , 2018, SIGDOC.

[4]  Hanspeter Pfister,et al.  What Makes a Visualization Memorable? , 2013, IEEE Transactions on Visualization and Computer Graphics.

[5]  Gwen Nugent,et al.  Deaf Students' Learning From Captioned Instruction: the Relationship Between the Visual and Caption Display , 1983 .

[6]  Aditya G. Parameswaran,et al.  S EE DB : Supporting Visual Analytics with Data-Driven Recommendations , 2015 .

[7]  M. Just,et al.  Constructing mental models of machines from text and diagrams. , 1993 .

[8]  Gitte Lindgaard,et al.  Improving accessibility to statistical graphs: the iGraph-Lite system , 2007, Assets '07.

[9]  Eunyee Koh,et al.  A Formative Study on Designing Accurate and Natural Figure Captioning Systems , 2020, CHI Extended Abstracts.

[10]  Qiang Zhang,et al.  TIARA: a visual exploratory text analytic system , 2010, KDD '10.

[11]  Jim Hunter,et al.  SumTime-Turbine: A Knowledge-Based System to Communicate Gas Turbine Time-Series Data , 2003, IEA/AIE.

[12]  Oded Nov,et al.  How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques , 2015, CHI.

[13]  J. Bransford.  Human Cognition: Learning, Understanding and Remembering , 1979 .

[14]  Maneesh Agrawala,et al.  Facilitating Document Reading by Linking Text and Tables , 2018, UIST.

[15]  Leland Wilkinson,et al.  AutoVis: Automatic Visualization , 2010, Inf. Vis..

[16]  Guy Lapalme,et al.  PostGraphe: A System for the Generation of Statistical Graphics and Text , 1996, INLG.

[17]  Kevin Zeng Hu,et al.  DIVE: A Mixed-Initiative System Supporting Integrated Data Exploration Workflows , 2018, HILDA@SIGMOD.

[18]  Karrie Karahalios,et al.  Trust and Recall of Information across Varying Degrees of Title-Visualization Misalignment , 2019, CHI.

[19]  S. Yantis,et al.  Visual attention: control, representation, and time course. , 1997, Annual review of psychology.

[20]  Michael J. Haass,et al.  Data Visualization Saliency Model: A Tool for Evaluating Abstract Data Visualizations , 2018, IEEE Transactions on Visualization and Computer Graphics.

[21]  Razvan C. Bunescu,et al.  Figure Captioning with Reasoning and Sequence-Level Training , 2019, ArXiv.

[22]  Nicholas Diakopoulos,et al.  Contextifier: automatic generation of annotated stock visualizations , 2013, CHI.

[23]  Shimei Pan,et al.  Interactive, topic-based visual text summarization and analysis , 2009, CIKM.

[24]  Alex Endert,et al.  Augmenting Visualizations with Interactive Data Facts to Facilitate Interpretation and Communication , 2019, IEEE Transactions on Visualization and Computer Graphics.

[25]  Stephanie Elzer Schwartz,et al.  Information graphics: an untapped resource for digital libraries , 2006, SIGIR.

[26]  Steven Franconeri,et al.  The Curse of Knowledge in Visual Data Communication , 2019, IEEE Transactions on Visualization and Computer Graphics.

[27]  Maneesh Agrawala,et al.  Graphical Overlays: Using Layered Elements to Aid Chart Reading , 2012, IEEE Transactions on Visualization and Computer Graphics.

[28]  Maneesh Agrawala,et al.  Extracting references between text and charts via crowdsourcing , 2014, CHI.

[29]  Alain Breuleux,et al.  Multimedia and Comprehension: The Relationship among Text, Animation and Captions , 1995, J. Am. Soc. Inf. Sci..

[30]  Maneesh Agrawala,et al.  Answering Questions about Charts and Generating Visual Explanations , 2020, CHI.

[31]  Hagit Shatkay,et al.  Extracting Figures and Captions from Scientific Publications , 2018, CIKM.

[32]  Ingrid Zukerman,et al.  Exploring and Exploiting the Limited Utility of Captions in Recognizing Intention in Information Graphics , 2005, ACL.

[33]  Karrie Karahalios,et al.  Frames and Slants in Titles of Visualizations on Controversial Topics , 2018, CHI.

[34]  Shimei Pan,et al.  TIARA: Interactive, Topic-Based Visual Text Summarization and Analysis , 2012, TIST.

[35]  Ben Shneiderman,et al.  Readings in information visualization - using vision to think , 1999 .

[36]  Ingrid Zukerman,et al.  The automated understanding of simple bar charts , 2011, Artif. Intell..

[37]  Daniel Afergan,et al.  Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability , 2016, IEEE Transactions on Visualization and Computer Graphics.

[38]  Razvan C. Bunescu,et al.  Neural caption generation over figures , 2019, UbiComp/ISWC Adjunct.

[39]  Peter J. Haas,et al.  Foresight: Rapid Data Exploration Through Guideposts , 2017, ArXiv.

[40]  Mao Lin Huang,et al.  Highlighting in Information Visualization: A Survey , 2010, 2010 14th International Conference Information Visualisation.