A Review of Visualization Assessment in Terms of User Performance and Experience

Traditionally, information visualization research tends to focus only on performance related metrics such as task completion time and correctness. However, a recent interest on goals such as memorability, engagement, and fun has also sparked. We propose a reflection on visualization assessment supported by a profound analysis of research that includes user studies of visualization tools and techniques. The motivation for this study is to explore the possible benefits of having a unified standardized metric for usability and user experience that can become a clearer quantitative model of both. Such a metric can provide visualization researchers and practitioners a more succinct and accurate way to assess their visualizations.

[1]  John T. Stasko,et al.  Visualizing Social Media Content with SentenTree , 2017, IEEE Transactions on Visualization and Computer Graphics.

[2]  Stephen G. Kobourov,et al.  Map‐based Visualizations Increase Recall Accuracy of Data , 2015, Comput. Graph. Forum.

[3]  John T. Stasko,et al.  DataMeadow: A Visual Canvas for Analysis of Large-Scale Multivariate Data , 2007, IEEE VAST.

[4]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[5]  Boudewijn P. F. Lelieveldt,et al.  CyteGuide: Visual Guidance for Hierarchical Single-Cell Analysis , 2018, IEEE Transactions on Visualization and Computer Graphics.

[6]  Paul Parsons,et al.  Assessing User Engagement in Information Visualization , 2017, CHI Extended Abstracts.

[7]  Hanspeter Pfister,et al.  Beyond Memorability: Visualization Recognition and Recall , 2016, IEEE Transactions on Visualization and Computer Graphics.

[8]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[9]  Michael J. McGuffin,et al.  Tracing Tuples Across Dimensions: A Comparison of Scatterplots and Parallel Coordinate Plots , 2012, Comput. Graph. Forum.

[10]  Min Zhu,et al.  AmbiguityVis: Visualization of Ambiguity in Graph Layouts , 2016, IEEE Transactions on Visualization and Computer Graphics.

[11]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[12]  Carl Gutwin,et al.  Useful junk?: the effects of visual embellishment on comprehension and memorability of charts , 2010, CHI.

[13]  Jean-Daniel Fekete,et al.  Storytelling in Information Visualizations: Does it Engage Users to Explore Data? , 2015, CHI.

[14]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[15]  Niklas Elmqvist,et al.  Perception of Animated Node‐Link Diagrams for Dynamic Graphs , 2012, Comput. Graph. Forum.

[16]  Charles Perin,et al.  Using Gap Charts to Visualize the Temporal Evolution of Ranks and Scores , 2016, IEEE Computer Graphics and Applications.

[17]  Yang Liu,et al.  Somewhere Over the Rainbow: An Empirical Assessment of Quantitative Colormaps , 2018, CHI.

[18]  Jean-Daniel Fekete,et al.  A Principled Way of Assessing Visualization Literacy , 2014, IEEE Transactions on Visualization and Computer Graphics.

[19]  Pierre Dragicevic,et al.  The Attraction Effect in Information Visualization , 2017, IEEE Transactions on Visualization and Computer Graphics.

[20]  Niklas Elmqvist,et al.  Patterns for visualization evaluation , 2012, BELIV '12.

[21]  Hansjörg Schmauder,et al.  An Evaluation of Visual Search Support in Maps , 2017, IEEE Transactions on Visualization and Computer Graphics.

[22]  Min Chen,et al.  Pathways for Theoretical Advances in Visualization , 2017, IEEE Computer Graphics and Applications.

[23]  Joseph F. Dumas,et al.  A Practical Guide to Usability Testing , 1993 .

[24]  Jian Zhao,et al.  Supporting Handoff in Asynchronous Collaborative Sensemaking Using Knowledge-Transfer Graphs , 2018, IEEE Transactions on Visualization and Computer Graphics.

[25]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[26]  Bo Fu,et al.  Eye tracking the user experience - An evaluation of ontology visualization techniques , 2016, Semantic Web.

[27]  Melanie Tory.  User Studies in Visualization: A Reflection on Methods , 2014, Handbook of Human Centric Visualization.

[28]  Jeffrey Heer,et al.  Animated Transitions in Statistical Data Graphics , 2007, IEEE Transactions on Visualization and Computer Graphics.

[29]  Chaomei Chen,et al.  Top 10 Unsolved Information Visualization Problems , 2005, IEEE Computer Graphics and Applications.

[30]  Alex Endert,et al.  Beyond Usability and Performance: A Review of User Experience-focused Evaluations in Visualization , 2016, BELIV '16.

[31]  Niklas Elmqvist,et al.  TimeMatrix: Analyzing Temporal Social Networks Using Interactive Matrix-Based Visualizations , 2010, Int. J. Hum. Comput. Interact..

[32]  Chris North,et al.  Toward measuring visualization insight , 2006, IEEE Computer Graphics and Applications.

[33]  Remo Aslak Burkhard,et al.  Tube Map Visualization: Evaluation of a Novel Knowledge Visualization Application for the Transfer of Knowledge in Long-Term Projects , 2005, J. Univers. Comput. Sci..

[34]  Michael Burch,et al.  Visual task solution strategies in tree diagrams , 2013, 2013 IEEE Pacific Visualization Symposium (PacificVis).

[35]  Niklas Elmqvist,et al.  ConceptVector: Text Visual Analytics via Interactive Lexicon Building Using Word Embedding , 2018, IEEE Transactions on Visualization and Computer Graphics.

[36]  Hui Zhang,et al.  iTTVis: Interactive Visualization of Table Tennis Data , 2018, IEEE Transactions on Visualization and Computer Graphics.

[37]  Yalong Yang,et al.  Many-to-Many Geographically-Embedded Flow Visualisation: An Evaluation , 2019, IEEE Transactions on Visualization and Computer Graphics.

[38]  H. Freud.  Emotional Design Why We Love Or Hate Everyday Things , 2016 .

[39]  Wouter Meulemans,et al.  Map LineUps: Effects of spatial structure on graphical inference , 2017, IEEE Transactions on Visualization and Computer Graphics.

[40]  Chris North,et al.  An insight-based methodology for evaluating bioinformatics visualizations , 2005, IEEE Transactions on Visualization and Computer Graphics.

[41]  Steven Franconeri,et al.  ISOTYPE Visualization: Working Memory, Performance, and Engagement with Pictographs , 2015, CHI.