Kori: Interactive Synthesis of Text and Charts in Data Documents

Charts go hand in hand with text to communicate complex data and are widely adopted in news articles, online blogs, and academic papers. They provide graphical summaries of the data, while text explains the message and context. However, synthesizing information across text and charts is difficult; it requires readers to frequently shift their attention. We investigated ways to support the tight coupling of text and charts in data documents. To understand their interplay, we analyzed the design space of chart-text references through news articles and scientific papers. Informed by the analysis, we developed a mixed-initiative interface enabling users to construct interactive references between text and charts. It leverages natural language processing to automatically suggest references as well as allows users to manually construct other references effortlessly. A user study complemented with algorithmic evaluation of the system suggests that the interface provides an effective way to compose interactive data documents.

[1]  Daniel J. Wigdor,et al.  DataInk: Direct and Creative Data-Oriented Drawing , 2018, CHI.

[2]  Zhicheng Liu,et al.  Leveraging Text-Chart Links to Support Authoring of Data-Driven Articles with VizFlow , 2021, CHI.

[3]  Tobias Höllerer,et al.  ChartAccent: Annotation for data-driven storytelling , 2017, 2017 IEEE Pacific Visualization Symposium (PacificVis).

[4]  Mira Dontcheva,et al.  Data-Driven Guides: Supporting Expressive Design for Information Graphics , 2017, IEEE Transactions on Visualization and Computer Graphics.

[5]  Arvind Satyanarayan,et al.  Vega-Lite: A Grammar of Interactive Graphics , 2018, IEEE Transactions on Visualization and Computer Graphics.

[6]  Maneesh Agrawala,et al.  Facilitating Document Reading by Linking Text and Tables , 2018, UIST.

[7]  Cristina Conati,et al.  Gaze-Driven Adaptive Interventions for Magazine-Style Narrative Visualizations , 2019, IEEE transactions on visualization and computer graphics.

[8]  Shahid Latif,et al.  Talking Realities: Audio Guides in Virtual Reality Visualizations , 2021, IEEE Computer Graphics and Applications.

[9]  Daniel A. Keim,et al.  VisJockey : Enriching Data Stories through Orchestrated Interactive Visualization , 2014 .

[10]  Alexander Lex,et al.  From Visual Exploration to Storytelling and Back Again , 2016, bioRxiv.

[11]  Evan M. Peck,et al.  The Curious Case of Combining Text and Visualization , 2019, EuroVis.

[12]  Bongshin Lee,et al.  Charticulator: Interactive Construction of Bespoke Chart Layouts , 2019, IEEE Transactions on Visualization and Computer Graphics.

[13]  Pat Hanrahan,et al.  Polaris: A System for Query, Analysis, and Visualization of Multidimensional Relational Databases , 2002, IEEE Trans. Vis. Comput. Graph..

[14]  Petra Isenberg,et al.  Interaction Techniques for Visual Exploration Using Embedded Word-Scale Visualizations , 2020, CHI.

[15]  Arvind Satyanarayan,et al.  Reactive Vega: A Streaming Dataflow Architecture for Declarative Interactive Visualization , 2016, IEEE Transactions on Visualization and Computer Graphics.

[16]  John Sweller,et al.  The Split-Attention Effect , 2011 .

[17]  John T. Stasko,et al.  Data Illustrator: Augmenting Vector Design Tools with Lazy Data Binding for Expressive Visualization Authoring , 2018, CHI.

[18]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[19]  Arvind Satyanarayan,et al.  Lyra: An Interactive Visualization Design Environment , 2014, Comput. Graph. Forum.

[20]  Fabian Beck,et al.  Authoring Combined Textual and Visual Descriptions of Graph Data , 2019, EuroVis.

[21]  HeerJeffrey,et al.  D3 Data-Driven Documents , 2011 .

[22]  Piotr Nowakowski,et al.  The Collage Authoring Environment , 2011, ICCS.

[23]  Kanit Wongsuphasawat,et al.  Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations , 2016, IEEE Transactions on Visualization and Computer Graphics.

[24]  Logan Fiorella,et al.  Principles for Reducing Extraneous Processing in Multimedia Learning: Coherence, Signaling, Redundancy, Spatial Contiguity and Temporal Contiguity Principles. , 2014 .

[25]  Jeffrey Heer,et al.  Idyll: A Markup Language for Authoring and Publishing Interactive Articles on the Web , 2018, UIST.

[26]  John T. Stasko,et al.  Natural Language Interfaces for Data Analysis with Visualization: Considering What Has and Could Be Asked , 2017, EuroVis.

[27]  John Sweller,et al.  Instructional Visualizations, Cognitive Load Theory, and Visuospatial Processing , 2019, Visuospatial Processing for Education in Health and Natural Sciences.

[28]  Tobias Höllerer,et al.  iVisDesigner: Expressive Interactive Design of Information Visualizations , 2014, IEEE Transactions on Visualization and Computer Graphics.

[29]  Hanspeter Pfister,et al.  DataToon: Drawing Dynamic Network Comics With Pen + Touch Interaction , 2019, CHI.

[30]  Niklas Elmqvist,et al.  Elastic Documents: Coupling Text and Tables through Contextual Visualizations for Enhanced Document Reading , 2019, IEEE Transactions on Visualization and Computer Graphics.

[31]  Xiaoru Yuan,et al.  Automatic Annotation Synchronizing with Textual Description for Visualization , 2020, CHI.

[32]  J. Sweller.  Implications of Cognitive Load Theory for Multimedia Learning , 2005, The Cambridge Handbook of Multimedia Learning.

[33]  John Sweller,et al.  The Split-Attention Principle in Multimedia Learning , 2021, The Cambridge Handbook of Multimedia Learning.

[34]  Ronald Metoyer,et al.  Linking and Layout: Exploring the Integration of Text and Visualization in Storytelling , 2019, Comput. Graph. Forum.

[35]  Fabian Beck,et al.  Word-Sized Graphics for Scientific Texts , 2017, IEEE Transactions on Visualization and Computer Graphics.

[36]  R. Kochhar.  Middle Class Fortunes in Western Europe , 2017 .

[37]  T. V. Landesberger,et al.  A Deeper Understanding of Visualization‐Text Interplay in Geographic Data‐driven Stories , 2021, Comput. Graph. Forum.

[38]  Bongshin Lee,et al.  Authoring Data-Driven Videos with DataClips , 2017, IEEE Transactions on Visualization and Computer Graphics.

[39]  Fabian Beck,et al.  VIS Author Profiles: Interactive Descriptions of Publication Records Combining Text and Visualization , 2019, IEEE Transactions on Visualization and Computer Graphics.

[40]  Jock D. Mackinlay,et al.  Storytelling: The Next Step for Visualization , 2013, Computer.

[41]  Alex Endert,et al.  Augmenting Visualizations with Interactive Data Facts to Facilitate Interpretation and Communication , 2019, IEEE Transactions on Visualization and Computer Graphics.

[42]  Alexander M. Rush,et al.  Challenges in Data-to-Document Generation , 2017, EMNLP.

[43]  Fabian Beck,et al.  Exploring Interactive Linking Between Text and Visualization , 2018, EuroVis.

[44]  Bongshin Lee,et al.  Timeline Storyteller: The Design & Deployment of an Interactive Authoring Tool for Expressive Timeline Narratives , 2018 .

[45]  Jock D. Mackinlay,et al.  Automating the design of graphical presentations of relational information , 1986, TOGS.

[46]  M. Sheelagh T. Carpendale,et al.  The Emerging Genre of Data Comics , 2017, IEEE Computer Graphics and Applications.

[47]  Cristina Conati,et al.  Understanding the effectiveness of adaptive guidance for narrative visualization: a gaze-based analysis , 2020, IUI.

[48]  N. Riche,et al.  DataSelfie: Empowering People to Design Personalized Visuals to Represent Their Data , 2019, CHI.

[49]  Maneesh Agrawala,et al.  Extracting references between text and charts via crowdsourcing , 2014, CHI.

[50]  Scott A. Hale,et al.  Viz-Blocks: Building Visualizations and Documents in the Browser , 2019, EuroVis.

[51]  Maneesh Agrawala,et al.  Answering Questions about Charts and Generating Visual Explanations , 2020, CHI.

[52]  Dieter Schmalstieg,et al.  Context-Preserving Visual Links , 2011, IEEE Transactions on Visualization and Computer Graphics.

[53]  M. Sheelagh T. Carpendale,et al.  More Than Telling a Story: Transforming Data into Visually Shared Stories , 2015, IEEE Computer Graphics and Applications.

[54]  Wied Ruijssenaars,et al.  Encyclopedia of the Sciences of Learning , 2012 .

[55]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[56]  Jeffrey Heer,et al.  Generalized selection via interactive query relaxation , 2008, CHI.

[57]  Arvind Satyanarayan,et al.  Authoring Narrative Visualizations with Ellipsis , 2014, Comput. Graph. Forum.

[58]  Hanspeter Pfister,et al.  Beyond Memorability: Visualization Recognition and Recall , 2016, IEEE Transactions on Visualization and Computer Graphics.

[59]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[60]  Jeffrey Heer,et al.  Narrative Visualization: Telling Stories with Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[61]  Tomas Mikolov,et al.  Advances in Pre-Training Distributed Word Representations , 2017, LREC.