Empirical Evaluations with Domain Experts

Over the past thirty years, the visualization community has developed theories and models to explain visualization as a technology that augments human cognition by enabling the efficient, accurate, and timely discovery of meaningful information in data. Along the way, practitioners have also debated theories and practices for visualization evaluation: How do we generate durable, reliable evidence that a visualization is effective? Interestingly, there is still no consensus in the visualization research community how to evaluate visualization methods. The goal of this chapter is to rise awareness of still open issues in the visualization evaluation and to discuss appropriate evaluations suitable for different visualization approaches. This includes user studies and best practices to conduct them but also other approaches for suitable evaluation of visualization. The chapter is structured as a moderated dialog of two visualization experts.

[1]  Robert Michael Kirby,et al.  Visualizing multivalued data from 2D incompressible flows using concepts from painting , 1999, VIS '99.

[2]  Ben Shneiderman,et al.  Apply or Die: On the Role and Assessment of Application Papers in Visualization , 2017, IEEE Computer Graphics and Applications.

[3]  David H. Laidlaw,et al.  A Case Study Using Visualization Interaction Logs and Insight Metrics to Understand How Analysts Arrive at Insights , 2016, IEEE Transactions on Visualization and Computer Graphics.

[4]  Gerik Scheuermann,et al.  Detection and Visualization of Closed Streamlines in Planar Flows , 2001, IEEE Trans. Vis. Comput. Graph..

[5]  Leslie M. Blaha,et al.  Parameter Space Visualization for Large-scale Datasets Using Parallel Coordinate Plots. , 2016 .

[6]  Victoria Interrante,et al.  User Studies: Why, How, and When? , 2003, IEEE Computer Graphics and Applications.

[7]  Joseph J. LaViola,et al.  CavePainting: a fully immersive 3D artistic medium and interactive experience , 2001, I3D '01.

[8]  Tamara Munzner,et al.  Bridging from Goals to Tasks with Design Study Analysis Reports , 2018, IEEE Transactions on Visualization and Computer Graphics.

[9]  J. Pell,et al.  Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. , 2004, BMJ.

[10]  Cullen D. Jackson,et al.  CAVE and fishtank virtual-reality displays: a qualitative and quantitative comparison , 2006, IEEE Transactions on Visualization and Computer Graphics.

[11]  Jarke J. van Wijk,et al.  Force‐Directed Edge Bundling for Graph Visualization , 2009, Comput. Graph. Forum.

[12]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[13]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[14]  Thomas Wischgoll,et al.  Vortex Visualization in Ultra Low Reynolds Number Insect Flight , 2011, IEEE Transactions on Visualization and Computer Graphics.

[15]  Tamara Munzner,et al.  Design Study Methodology: Reflections from the Trenches and the Stacks , 2012, IEEE Transactions on Visualization and Computer Graphics.

[16]  Denis Gracanin,et al.  Interactive Visual Steering - Rapid Visual Prototyping of a Common Rail Injection System , 2008, IEEE Transactions on Visualization and Computer Graphics.

[17]  David H. Laidlaw,et al.  The application visualization system: a computational environment for scientific visualization , 1989, IEEE Computer Graphics and Applications.

[18]  Robert Michael Kirby,et al.  Comparing 2D vector field visualization methods: a user study , 2005, IEEE Transactions on Visualization and Computer Graphics.

[19]  Hans Hagen,et al.  Accurate and reliable extraction of surfaces from image data using a multi-dimensional uncertainty model , 2018, Graph. Model..

[20]  David H. Laidlaw,et al.  An insight- and task-based methodology for evaluating spatiotemporal visual analytics , 2014, 2014 IEEE Conference on Visual Analytics Science and Technology (VAST).

[21]  David H. Laidlaw,et al.  Experiments in Immersive Virtual Reality for Scientific Visualization , 2002, Comput. Graph..

[22]  Tamara Munzner,et al.  The four-level nested model revisited: blocks and guidelines , 2012, BELIV '12.

[23]  John Flach,et al.  Display Infrastructure for Virtual Environments , 2018, Visualization and Data Analysis.

[24]  Tobias Isenberg,et al.  A Systematic Review on the Practice of Evaluating Visualization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[25]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[26]  David S. Ebert,et al.  An Ontological Framework for Supporting the Design and Evaluation of Visual Analytics Systems , 2019, Comput. Graph. Forum.