EvalBench: A Software Library for Visualization Evaluation

It is generally acknowledged in visualization research that it is necessary to evaluate visualization artifacts in order to provide empirical evidence on their effectiveness and efficiency as well as their usability and utility. However, the difficulties of conducting such evaluations still remain an issue. Apart from the required know‐how to appropriately design and conduct user studies, the necessary implementation effort for evaluation features in visualization software is a considerable obstacle. To mitigate this, we present EvalBench, an easy‐to‐use, flexible, and reusable software library for visualization evaluation written in Java. We describe its design choices and basic abstractions of our conceptual architecture and demonstrate its applicability by a number of case studies. EvalBench reduces implementation effort for evaluation features and makes conducting user studies easier. It can be used and integrated with third‐party visualization prototypes that need to be evaluated via loose coupling. EvalBench supports both, quantitative and qualitative evaluation methods such as controlled experiments, interaction logging, laboratory questionnaires, heuristic evaluations, and insight diaries.

[1]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[2]  Yuval Shahar,et al.  Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data , 2008, Artif. Intell. Medicine.

[3]  Chris North,et al.  Toward measuring visualization insight , 2006, IEEE Computer Graphics and Applications.

[4]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[5]  Silvia Miksch,et al.  Patient Development at a Glance: An Evaluation of a Medical Data Visualization , 2011, INTERACT.

[6]  Keith Andrews,et al.  A Comparative Study of Four Hierarchy Browsers using the Hierarchical Visualisation Testing Environment (HVTE) , 2007, 2007 11th International Conference Information Visualization (IV '07).

[7]  Jarke J. van Wijk,et al.  Supporting the analytical reasoning process in information visualization , 2008, CHI.

[8]  Jereme Haack,et al.  Glass box: capturing, archiving, and retrieving workstation activities , 2006, CARPE '06.

[9]  Jeffrey Heer,et al.  SpanningAspectRatioBank Easing FunctionS ArrayIn ColorIn Date Interpolator MatrixInterpola NumObjecPointI Rectang ISchedu Parallel Pause Scheduler Sequen Transition Transitioner Transiti Tween Co DelimGraphMLCon IData JSONCon DataField DataSc Dat DataSource Data DataUtil DirtySprite LineS RectSprite , 2011 .

[10]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[11]  Jean-Daniel Fekete,et al.  Obvious: A meta-toolkit to encapsulate information visualization toolkits — One toolkit to bind them all , 2011, 2011 IEEE Conference on Visual Analytics Science and Technology (VAST).

[12]  Stefan Biffl,et al.  PlanningLines: novel glyphs for representing temporal uncertainties and their evaluation , 2005, Ninth International Conference on Information Visualisation (IV'05).

[13]  Chris North,et al.  Information Visualization , 2008, Lecture Notes in Computer Science.

[14]  I. Scott MacKenzie,et al.  Generalized Fitts' law model builder , 1995, CHI '95.

[15]  Silvia Miksch,et al.  Bertin was Right: An Empirical Evaluation of Indexing to Compare Multivariate Time‐Series Data Using Line Plots , 2011, Comput. Graph. Forum.

[16]  Daniel A. Keim,et al.  Mastering the Information Age - Solving Problems with Visual Analytics , 2010 .

[17]  Camilla Forsell,et al.  A Guide to Scientific Evaluation in Information Visualization , 2010, 2010 14th International Conference Information Visualisation.

[18]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[19]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[20]  Niklas Elmqvist,et al.  Patterns for visualization evaluation , 2012, BELIV '12.

[21]  Marie desJardins,et al.  An animated multivariate visualization for physiological and clinical data in the ICU , 2010, IHI.

[22]  Jing Jin,et al.  Interactive querying of temporal data using a comic strip metaphor , 2010, 2010 IEEE Symposium on Visual Analytics Science and Technology.

[23]  Silvia Miksch,et al.  Analysing Interactivity in Information Visualisation , 2012, KI - Künstliche Intelligenz.

[24]  Héctor Lafaille.  M.A.R. , 1957 .

[25]  Suziah Sulaiman,et al.  Usability Management System (USEMATE): A web-based automated system for managing usability testing systematically , 2010, 2010 International Conference on User Science and Engineering (i-USEr).

[26]  Olivier Chapuis,et al.  Touchstone: exploratory design of experiments , 2007, CHI.

[27]  Silvia Miksch,et al.  Evaluating an InfoVis Technique Using Insight Reports , 2007, 2007 11th International Conference Information Visualization (IV '07).

[28]  Cláudio T. Silva,et al.  VisTrails: visualization meets data management , 2006, SIGMOD Conference.

[29]  Jeffrey Heer,et al.  prefuse: a toolkit for interactive information visualization , 2005, CHI.

[30]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[31]  Randolph G. Bias,et al.  Research Methods for Human-Computer Interaction , 2010, J. Assoc. Inf. Sci. Technol..

[32]  Jan Theeuwes,et al.  OpenSesame: An open-source, graphical experiment builder for the social sciences , 2011, Behavior Research Methods.

[33]  Bruce Phillips,et al.  Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems , 2008, CHI.

[34]  David H. Laidlaw,et al.  Modeling task performance for a crowd of users from interaction histories , 2012, CHI.

[35]  John T. Stasko,et al.  Toward a Deeper Understanding of the Role of Interaction in Information Visualization , 2007, IEEE Transactions on Visualization and Computer Graphics.

[36]  Jeffrey Heer,et al.  Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation , 2008, IEEE Transactions on Visualization and Computer Graphics.

[37]  Jean-Daniel Fekete,et al.  A Methodological Note on Setting-up Logging and Replay Mechanisms in InfoVis Systems , 2008 .

[38]  William Ribarsky,et al.  Recovering Reasoning Processes from User Interactions , 2009, IEEE Computer Graphics and Applications.

[39]  Wolfgang Aigner,et al.  Comparative Evaluation of an Interactive Time‐Series Visualization that Combines Quantitative Data with Qualitative Abstractions , 2012, Comput. Graph. Forum.