Patterns for visualization evaluation

We propose a patterns-based approach to evaluating data visualization: a set of general and reusable solutions to commonly occurring problems in evaluating tools, techniques, and systems for visual sensemaking. Patterns have had significant impact in a wide array of disciplines, particularly software engineering, and we believe that they provide a powerful lens for looking at visualization evaluation by offering practical, tried-and-tested tips and tricks that can be adopted immediately. The 12 patterns presented here have also been added to a freely editable Wiki repository. The motivation for creating this evaluation pattern language is to (a) disseminate hard-won experience on visualization evaluation to researchers and practitioners alike; to (b) provide a standardized vocabulary for designing visualization evaluation; and to (c) invite the community to add new evaluation patterns to a growing repository of patterns.

[1]  Aniket Kittur,et al.  An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets , 2011, ICWSM.

[2]  Melanie Tory,et al.  Evaluating Visualizations: Do Expert Reviews Work? , 2005, IEEE Computer Graphics and Applications.

[3]  Jean-Daniel Fekete,et al.  MatrixExplorer: a Dual-Representation System to Explore Social Networks , 2006, IEEE Transactions on Visualization and Computer Graphics.

[4]  Jeffrey Heer,et al.  Animated Transitions in Statistical Data Graphics , 2007, IEEE Transactions on Visualization and Computer Graphics.

[5]  Pierre Dragicevic,et al.  Color Lens: Adaptive Color Scale Optimization for Visual Exploration , 2011, IEEE Transactions on Visualization and Computer Graphics.

[6]  Max Jacobson,et al.  A Pattern Language: Towns, Buildings, Construction , 1981 .

[7]  Andrew Koenig,et al.  Patterns and Antipatterns , 1998, J. Object Oriented Program..

[8]  Paul Johns,et al.  Understanding Pen and Touch Interaction for Data Exploration on Interactive Whiteboards , 2012, IEEE Transactions on Visualization and Computer Graphics.

[9]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[10]  Pierre Dragicevic,et al.  Rolling the Dice: Multidimensional Visual Exploration using Scatterplot Matrix Navigation , 2008, IEEE Transactions on Visualization and Computer Graphics.

[11]  Tamara Munzner,et al.  An evaluation of pan & zoom and rubber sheet navigation with and without an overview , 2006, CHI.

[12]  Ahmed Sabbir Arif,et al.  Analysis of text entry performance metrics , 2009, 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH).

[13]  Jean Scholtz,et al.  Evaluating Visual Analytics at the 2007 VAST Symposium Contest , 2008, IEEE Computer Graphics and Applications.

[14]  Chris North,et al.  A Comparison of User-Generated and Automatic Graph Layouts , 2009, IEEE Transactions on Visualization and Computer Graphics.

[15]  John T. Stasko,et al.  Jigsaw: Supporting Investigative Analysis through Interactive Visualization , 2007, 2007 IEEE Symposium on Visual Analytics Science and Technology.

[16]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[17]  Tamara Munzner,et al.  Overview Use in Multiple Visual Information Resolution Interfaces , 2007, IEEE Transactions on Visualization and Computer Graphics.

[18]  Gerald Penn,et al.  Living with a Tabletop: Analysis and Observations of Long Term Office Use of a Multi-Touch Table , 2007, Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP'07).

[19]  T. Marteau,et al.  The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study , 1997 .

[20]  Pierre Dragicevic,et al.  Temporal distortion for animated transitions , 2011, CHI.

[21]  Sung-Hee Kim,et al.  How to filter out random clickers in a crowdsourcing-based study? , 2012, BELIV '12.

[22]  Tamara Munzner,et al.  Process and Pitfalls in Writing Information Visualization Research Papers , 2008, Information Visualization.

[23]  John T. Stasko,et al.  Effectiveness of Animation in Trend Visualization , 2008, IEEE Transactions on Visualization and Computer Graphics.

[24]  Robert Kosara,et al.  Do Mechanical Turks dream of square pie charts? , 2010, BELIV '10.

[25]  Brian R. Gaines,et al.  Modeling and forecasting the information sciences , 1991, Inf. Sci..

[26]  John T. Stasko,et al.  Dust & Magnet: Multivariate Information Visualization Using a Magnet Metaphor , 2005, Inf. Vis..

[27]  Gordon Bell,et al.  MyLifeBits: a personal database for everything , 2006, CACM.

[28]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[29]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[30]  John T. Stasko,et al.  SellTrend: Inter-Attribute Visual Analysis of Temporal Transaction Data , 2009, IEEE Transactions on Visualization and Computer Graphics.

[31]  M. Sheelagh T. Carpendale,et al.  Grounded evaluation of information visualizations , 2008, BELIV.

[32]  John Stasko.  Evaluating Information Visualizations : Issues and Opportunities , 2006 .

[33]  James A. Landay,et al.  SketchWizard: Wizard of Oz prototyping of pen-based user interfaces , 2007, UIST '07.

[34]  Ken Brodlie,et al.  GRASPARC-A problem solving environment integrating computation and visualization , 1993, Proceedings Visualization '93.

[35]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[36]  P M Boynton.  People should participate in, not be subjects of, research , 1998, BMJ.

[37]  D. Weiss,et al.  Interrater reliability and agreement of subjective judgments , 1975 .

[38]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[39]  David S. Ebert,et al.  Evaluating the Role of Time in Investigative Analysis of Document Collections , 2012, IEEE Transactions on Visualization and Computer Graphics.

[40]  Heidi Lam,et al.  A Framework of Interaction Costs in Information Visualization , 2008, IEEE Transactions on Visualization and Computer Graphics.

[41]  Niklas Elmqvist,et al.  Polyzoom: multiscale and multifocus exploration in 2d visual spaces , 2012, CHI.

[42]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[43]  John T. Stasko,et al.  Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study , 2009, 2009 IEEE Symposium on Visual Analytics Science and Technology.

[44]  Christopher Andrews,et al.  Space to think: large high-resolution displays for sensemaking , 2010, CHI.

[45]  John Hardy.  Experiences: a year in the life of an interactive desk , 2012, DIS '12.

[46]  Arne Jönsson,et al.  Wizard of Oz studies -- why and how , 1993, Knowl. Based Syst..

[47]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[48]  Melanie Tory,et al.  How Information Visualization Novices Construct Visualizations , 2010, IEEE Trans. Vis. Comput. Graph..

[49]  Emmanuel Pietriga,et al.  Pointing and beyond: an operationalization and preliminary evaluation of multi-scale searching , 2007, CHI.

[50]  Niklas Elmqvist,et al.  DataMeadow: A Visual Canvas for Analysis of Large-Scale Multivariate Data , 2007 .

[51]  Erich Gamma.  Design Patterns - Past, Present & Future , 2010, The Future of Software Engineering.

[52]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[53]  Jeffrey Heer,et al.  Sizing the horizon: the effects of chart size and layering on the graphical perception of time series visualizations , 2009, CHI.

[54]  M. Sheelagh T. Carpendale,et al.  An exploratory study of visual information analysis , 2008, CHI.

[55]  Jeffrey Heer,et al.  Software Design Patterns for Information Visualization , 2006, IEEE Transactions on Visualization and Computer Graphics.

[56]  Niklas Elmqvist,et al.  TimeMatrix: Analyzing Temporal Social Networks Using Interactive Matrix-Based Visualizations , 2010, Int. J. Hum. Comput. Interact..

[57]  Chris Callison-Burch,et al.  Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk , 2009, EMNLP.

[58]  J. Creswell.  Qualitative inquiry and research design: choosing among five traditions. , 1998 .

[59]  Colin Ware,et al.  Cognitive Measurements of Graph Aesthetics , 2002, Inf. Vis..

[60]  Lane Harrison,et al.  Towards a 3-dimensional model of individual cognitive differences: position paper , 2012, BELIV '12.

[61]  Sung-Hee Kim,et al.  Does an Eye Tracker Tell the Truth about Visualizations?: Findings while Investigating Visualizations for Decision Making , 2012, IEEE Transactions on Visualization and Computer Graphics.

[62]  Chris North,et al.  Toward measuring visualization insight , 2006, IEEE Computer Graphics and Applications.

[63]  Yvonne Rogers,et al.  Interaction Design: Beyond Human-Computer Interaction , 2002 .

[64]  Niklas Elmqvist,et al.  Evaluating Physical/Virtual Occlusion Management Techniques for Horizontal Displays , 2011, INTERACT.

[65]  Chris North,et al.  An Insight-Based Longitudinal Study of Visual Analytics , 2006, IEEE Transactions on Visualization and Computer Graphics.

[66]  William Ribarsky,et al.  Evaluating the relationship between user interaction and financial visual analysis , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[67]  Staffan Björk,et al.  Patterns In Game Design , 2004 .

[68]  Stuart K. Card,et al.  Entity-based collaboration tools for intelligence analysis , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[69]  Anthony C. Robinson,et al.  Collaborative synthesis of visual analytic results , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[70]  Niklas Elmqvist,et al.  Improving revisitation in graphs through static spatial features , 2011, Graphics Interface.

[71]  Niklas Elmqvist,et al.  Perception of Animated Node‐Link Diagrams for Dynamic Graphs , 2012, Comput. Graph. Forum.

[72]  Jean-Daniel Fekete,et al.  Melange: space folding for multi-focus interaction , 2008, CHI.

[73]  Brian D. Fisher,et al.  Pair Analytics: Capturing Reasoning Processes in Collaborative Visual Analytics , 2011, 2011 44th Hawaii International Conference on System Sciences.

[74]  Chris North,et al.  Information Visualization , 2008, Lecture Notes in Computer Science.

[75]  Bernice E. Rogowitz,et al.  Perceptual Organization in User-Generated Graph Layouts , 2008, IEEE Transactions on Visualization and Computer Graphics.

[76]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.