Toward visualization-specific heuristic evaluation

This position paper describes heuristic evaluation as it relates to visualization and visual analytics. We review heuristic evaluation in general, then comment on previous process-based, performance-based, and framework-based efforts to adapt the method to visualization-specific needs. We postulate that the framework-based approach holds the most promise for future progress in development of visualization-specific heuristics, and propose a specific framework as a starting point. We then recommend a method for community involvement and input into the further development of the heuristic framework and more detailed design and evaluation guidelines.

[1]  Jock D. Mackinlay,et al.  Automating the design of graphical presentations of relational information , 1986, TOGS.

[2]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[3]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[4]  Bruce Tognazzini,et al.  Tog on Interface , 1992 .

[5]  Jade Goldstein-Stewart,et al.  Interactive graphic design using automatic presentation knowledge , 1994, CHI '94.

[6]  J. Kitzinger,et al.  Qualitative Research: Introducing focus groups , 1995 .

[7]  Michael J. Wenger,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests by Jeffrey Rubin 330 pages, $34.95 New York: John Wiley & Sons, 1994 ISBN 0-471-59403-2 , 1995 .

[8]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[9]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[10]  Jill Gerhardt‐Powals,et al.  Cognitive engineering principles for enhancing human-computer performance , 1996, Int. J. Hum. Comput. Interact..

[11]  Wilbert O. Galitz,et al.  The Essential Guide to User Interface Design: An Introduction to GUI Design Principles and Techniques , 1996 .

[12]  Dominique L. Scapin,et al.  Ergonomic criteria for evaluating the ergonomic quality of interactive systems , 1997, Behav. Inf. Technol..

[13]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[14]  Joseph M. Hellerstein,et al.  Potter's Wheel: An Interactive Data Cleaning System , 2001, VLDB.

[15]  Marti A. Hearst,et al.  The state of the art in automating usability evaluation of user interfaces , 2001, CSUR.

[16]  Marco Winckler,et al.  On evaluating information visualization techniques , 2002, AVI '02.

[17]  A. Lancaster,et al.  Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces , 2004, IEEE Transactions on Professional Communication.

[18]  John T. Stasko,et al.  BEST PAPER: A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations , 2004, IEEE Symposium on Information Visualization.

[19]  Melanie Tory,et al.  Evaluating Visualizations: Do Expert Reviews Work? , 2005, IEEE Computer Graphics and Applications.

[20]  Catherine Courage,et al.  CHAPTER 1 – INTRODUCTION TO USER REQUIREMENTS , 2005 .

[21]  Paul A. Cairns,et al.  Beyond guidelines: what can we learn from the visual information seeking mantra? , 2005, Ninth International Conference on Information Visualisation (IV'05).

[22]  Catherine Courage,et al.  Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and Techniques , 2005 .

[23]  M. Sheelagh T. Carpendale,et al.  Theoretical analysis of uncertainty visualizations , 2006, Electronic Imaging.

[24]  M. Sheelagh T. Carpendale,et al.  Heuristics for information visualization evaluation , 2006, BELIV '06.

[25]  Richard Buchanan,et al.  Understanding Your Users: A Practical Guide to User Requirements: Methods, Tools, and Techniques by Catherine Courage and Kathy Baxter , ISBN 1558609350 ( San Francisco : Morgan Kaufmann , 2005 ) 781 pages; $59.95; paperback. , 2007, Design Issues.

[26]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[27]  Chris North,et al.  Information Visualization , 2008, Lecture Notes in Computer Science.

[28]  Xing Xie,et al.  Mining interesting locations and travel sequences from GPS trajectories , 2009, WWW '09.

[29]  William Ribarsky,et al.  Recovering Reasoning Processes from User Interactions , 2009, IEEE Computer Graphics and Applications.

[30]  Michelle X. Zhou,et al.  Characterizing Users' Visual Analytic Activity for Insight Provenance , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[31]  Robert Spence,et al.  The Broker , 2009, HCIV.

[32]  Chauncey E. Wilson,et al.  User Experience Re-Mastered: Your Guide to Getting the Right Design , 2009 .

[33]  Jarke J. van Wijk,et al.  Supporting Exploration Awareness in Information Visualization , 2009, IEEE Computer Graphics and Applications.

[34]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[35]  Andreas Butz,et al.  Evaluating information visualization in large companies: challenges, experiences and recommendations , 2010, BELIV '10.

[36]  Camilla Forsell,et al.  An heuristic set for evaluation in information visualization , 2010, AVI.

[37]  Randolph G. Bias,et al.  Research Methods for Human-Computer Interaction , 2010, J. Assoc. Inf. Sci. Technol..

[38]  Natalie J Corrao,et al.  Importance of testing for usability when selecting and implementing an electronic health or medical record system. , 2010, Journal of oncology practice.

[39]  Jean Scholtz.  Developing qualitative metrics for visual analytic environments , 2010, BELIV '10.

[40]  Jean Scholtz,et al.  Developing guidelines for assessing visual analytics environments , 2011, Inf. Vis..

[41]  J. Ivey.  Focus groups. , 2011, Pediatric nursing.

[42]  Günter Wallner,et al.  Human Centered Design in Practice: A Case Study with the Ontology Visualization Tool Knoocks , 2011, ICCV 2011.

[43]  Silvia Miksch,et al.  Analysing Interactivity in Information Visualisation , 2012, KI - Künstliche Intelligenz.

[44]  Rex Hartson,et al.  The UX book, process and guidelines for ensuring a quality user experience by Rex Hartson and Pardha S. Pyla , 2012, SOEN.

[45]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[46]  Camilla Forsell.  Evaluation in Information Visualization: Heuristic Evaluation , 2012, 2012 16th International Conference on Information Visualisation.

[47]  Silvia Miksch,et al.  A Taxonomy of Dirty Time-Oriented Data , 2012, CD-ARES.

[48]  Emily S. Patterson,et al.  Technical evaluation, testing, and validatiaon of the usability of electronic health records , 2012 .

[49]  Tamara Munzner,et al.  The four-level nested model revisited: blocks and guidelines , 2012, BELIV '12.

[50]  Jeffrey Heer,et al.  Profiler: integrated statistical analysis and visualization for data quality assessment , 2012, AVI.

[51]  PlaisantCatherine,et al.  Empirical Studies in Information Visualization , 2012 .

[52]  Nathan Franklin,et al.  The UX book: Process and guidelines for ensuring a quality user experience by Rex Hartson and Pardha A. Pyla, San Diego: Morgan Kaufmann. 2012. , 2013, CDQR.

[53]  Tobias Isenberg,et al.  A Systematic Review on the Practice of Evaluating Visualization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[54]  Harvey P. Siy,et al.  The Chicken and the Pig: User Involvement in Developing Usability Heuristics , 2013, AMCIS.

[55]  Dominique L. Scapin,et al.  User-Centered Evaluation of Information Visualization Techniques: Making the HCI-InfoVis Connection Explicit , 2014, Handbook of Human Centric Visualization.

[56]  Silvia Miksch,et al.  TimeCleanser: a visual analytics approach for data cleansing of time-oriented data , 2014, i-KNOW '14.

[57]  Paul R. Havig,et al.  A human cognition framework for information visualization , 2014, Comput. Graph..

[58]  Daniel A. Keim,et al.  A Visual Analytics Field Experiment to Evaluate Alternative Visualizations for Cyber Security Applications , 2014, EuroVA@EuroVis.

[59]  Melanie Tory.  User Studies in Visualization: A Reflection on Methods , 2014, Handbook of Human Centric Visualization.

[60]  Julia Kastner,et al.  Show Me The Numbers Designing Tables And Graphs To Enlighten , 2016 .