Reflecting on the Evaluation of Visualization Authoring Systems : Position Paper

In this paper, we discuss the challenges one faces when evaluating authoring systems developed to help people design visualization for communication purposes. We reflect on our own experiences in evaluating the visualization authoring systems that we have developed as well as the evaluation methods used in other recent projects. We also examine alternative approaches for evaluating visualization authoring systems that we believe to be more appropriate than traditional comparative studies. We hope that our discussion is informative, not only for researchers who intend to develop novel visualization authoring systems, but also for reviewers assigned to evaluate the research contributions of these systems. Our discussion concludes with opportunities for facilitating the evaluation and adoption of deployed visualization authoring systems.

[1]  Bongshin Lee,et al.  Authoring Data-Driven Videos with DataClips , 2017, IEEE Transactions on Visualization and Computer Graphics.

[2]  Miguel A. Nacenta,et al.  iVoLVER: Interactive Visual Language for Visualization Extraction and Reconstruction , 2016, CHI.

[3]  Wei Chen,et al.  The design space of construction tools for information visualization: A survey , 2017, J. Vis. Lang. Comput..

[4]  Tobias Isenberg,et al.  Visualization as Seen through its Research Paper Keywords , 2014, IEEE Transactions on Visualization and Computer Graphics.

[5]  Tobias Isenberg,et al.  Visualization as Seen through its Research Paper Keywords , 2014, IEEE Transactions on Visualization and Computer Graphics.

[6]  Xi Chen,et al.  InfoNice: Easy Creation of Information Graphics , 2018, CHI.

[7]  Tamara Munzner,et al.  Design Study Methodology: Reflections from the Trenches and the Stacks , 2012, IEEE Transactions on Visualization and Computer Graphics.

[8]  Jeffrey Heer,et al.  Narrative Visualization: Telling Stories with Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[9]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[10]  Benjamin Wiederkehr,et al.  Evaluating Data-Driven Stories and Storytelling Tools * , 2018 .

[11]  John T. Stasko,et al.  Data Illustrator: Augmenting Vector Design Tools with Lazy Data Binding for Expressive Visualization Authoring , 2018, CHI.

[12]  Daniel J. Wigdor,et al.  DataInk: Direct and Creative Data-Oriented Drawing , 2018, CHI.

[13]  Steven M. Drucker,et al.  Iterating between Tools to Create and Edit Visualizations , 2017, IEEE Transactions on Visualization and Computer Graphics.

[14]  Uta Hinrichs,et al.  Bottom-up vs. Top-down: Trade-offs in Efficiency, Understanding, Freedom and Creativity with InfoVis Tools , 2017, CHI.

[15]  Hans-Jörg Schulz,et al.  Treevis.net: A Tree Visualization Reference , 2011, IEEE Computer Graphics and Applications.

[16]  Bongshin Lee,et al.  Visual Narrative Flow: Exploring Factors Shaping Data Visualization Story Reading Experiences , 2017, Comput. Graph. Forum.

[17]  Fereshteh Amini,et al.  Evaluating Data-Driven Stories & Storytelling Tools , 2018 .

[18]  Martin Wattenberg,et al.  Harry Potter and the Meat-Filled Freezer: A Case Study of Spontaneous Usage  of Visualization Tools , 2008, Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008).

[19]  Bongshin Lee,et al.  Charticulator: Interactive Construction of Bespoke Chart Layouts , 2019, IEEE Transactions on Visualization and Computer Graphics.

[20]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[21]  Christophe Hurter,et al.  Understanding Data Videos: Looking at Narrative Visualization through the Cinematography Lens , 2015, CHI.

[22]  Wei Chen,et al.  VisComposer: A Visual Programmable Composition Environment for Information Visualization , 2018, Vis. Informatics.

[23]  M. Sheelagh T. Carpendale,et al.  Pre-design empiricism for information visualization: scenarios, methods, and challenges , 2014, BELIV '14.

[24]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[25]  Bongshin Lee,et al.  Data-Driven Storytelling Techniques : Analysis of a Curated Collection of Visual Stories , 2018 .

[26]  Tamara Munzner,et al.  Overview: The Design, Adoption, and Analysis of a Visual Document Mining Tool for Investigative Journalists , 2014, IEEE Transactions on Visualization and Computer Graphics.

[27]  Melanie Tory,et al.  A Survey of Visualization Construction User Interfaces , 2013, EuroVis.

[28]  Allen Newell,et al.  The keystroke-level model for user performance time with interactive systems , 1980, CACM.

[29]  Tobias Höllerer,et al.  iVisDesigner: Expressive Interactive Design of Information Visualizations , 2014, IEEE Transactions on Visualization and Computer Graphics.

[30]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[31]  Daniel F. Keefe,et al.  Visualization-by-Sketching: An Artist's Interface for Creating Multivariate Time-Varying Data Visualizations , 2016, IEEE Transactions on Visualization and Computer Graphics.

[32]  Mary Czerwinski,et al.  Selected Human Factors Issues in Information Visualization , 2009 .

[33]  Tobias Höllerer,et al.  ChartAccent: Annotation for data-driven storytelling , 2017, 2017 IEEE Pacific Visualization Symposium (PacificVis).

[34]  Martin Wattenberg,et al.  ManyEyes: a Site for Visualization at Internet Scale , 2007, IEEE Transactions on Visualization and Computer Graphics.

[35]  Arvind Satyanarayan,et al.  Lyra: An Interactive Visualization Design Environment , 2014, Comput. Graph. Forum.

[36]  Martin Wattenberg,et al.  Your place or mine?: visualization as a community component , 2008, CHI.

[37]  T. Munzner,et al.  Timelines Revisited: A Design Space and Considerations for Expressive Storytelling. , 2016, IEEE transactions on visualization and computer graphics.

[38]  Arvind Satyanarayan,et al.  Authoring Narrative Visualizations with Ellipsis , 2014, Comput. Graph. Forum.