Augmenting Visualization with Natural Language Translation of Interaction: A Usability Study

As visualization tools get more complicated, users often find it increasingly difficult to learn interaction sequences, recall past queries, and interpret visual states. We examine a query‐to‐question (Q2Q) supporting system that takes advantage of natural language generation (NLG) techniques to automatically translate and display query interactions as natural language questions. We focus on a symmetric pattern of multiple coordinated views, cross‐filtered views, that involves only nominal/categorical data. We describe a study of the effects of pairing a visualization with a Q2Q interface on several aspects of usability. Q2Q produces considerable improvements in learnability, efficiency, and memorability of visualization in terms of speed and the length of interaction sequences that users follow, along with a modest decrease in error ratio. From a visual language perspective, we analyze how Q2Q speeds up users’ comprehension of interaction, particularly when a visualization representation has deficiencies in illustrating hidden items or relationships.

[1]  Miron Livny,et al.  Improvise: a user interface for interactive construction of highly-coordinated visualizations , 2006 .

[2]  Heidrun Schumann,et al.  A History Mechanism for Visual Data Mining , 2004, IEEE Symposium on Information Visualization.

[3]  James Shaw,et al.  Practical Issues in Automatic Documentation Generation , 1994, ANLP.

[4]  Michelle X. Zhou,et al.  Characterizing Users' Visual Analytic Activity for Insight Provenance , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[5]  Chris Weaver,et al.  Cross-Filtered Views for Multidimensional Visual Analysis , 2010, IEEE Transactions on Visualization and Computer Graphics.

[6]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[7]  N. Hari Narayanan,et al.  Visual language theory: towards a human computer interaction perspective , 1998 .

[8]  Robert Harper,et al.  Stories in GeoTime , 2007, 2007 IEEE Symposium on Visual Analytics Science and Technology.

[9]  Johanna D. Moore,et al.  Describing Complex Charts in Natural Language: A Caption Generation System , 1998, CL.

[10]  Peter Brusilovsky.  Program visualization as a debugging tool for novices , 1993, CHI '93.

[11]  Niko Myller.  Automatic Generation of Prediction Questions during Program Visualization , 2007, Electron. Notes Theor. Comput. Sci..

[12]  Thomas L. Naps.  JHAVÉ – Addressing the Need to Support Algorithm Visualization with Tools for Active Engagement , 2005 .

[13]  Cláudio T. Silva,et al.  Provenance for Visualizations: Reproducibility and Beyond , 2007, Computing in Science & Engineering.

[14]  Gregory D. Abowd,et al.  Integrating status and event phenomena in formal specifications of interactive systems , 1994, SIGSOFT '94.

[15]  William Ribarsky,et al.  Evaluating the relationship between user interaction and financial visual analysis , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[16]  Peter Brusilovsky,et al.  WADEIn II: a case for adaptive explanatory visualization , 2006, ITICSE '06.

[17]  Jim Hunter,et al.  Summarizing Neonatal Time Series Data , 2003, EACL.

[18]  Thomas L. Naps.  JHAVE: supporting algorithm visualization , 2005, IEEE Computer Graphics and Applications.

[19]  Amruth N. Kumar.  Model-Based Generation of Demand Feedback in a Programming Tutor , 2003 .

[20]  Maryam Nafari,et al.  Poster: Translating cross-filtered queries into questions , 2010, 2010 IEEE Symposium on Visual Analytics Science and Technology.

[21]  Peter Brusilovsky,et al.  Explanatory Visualization in an Educational Programming Environment: Connecting Examples with General Knowledge , 1994, EWHCI.

[22]  Robert Dale,et al.  Building Natural Language Generation Systems: Figures , 2000 .

[23]  Martin Wattenberg,et al.  Voyagers and voyeurs: supporting asynchronous collaborative information visualization , 2007, CHI.

[24]  Amruth N. Kumar,et al.  A tutor for counter-controlled loop concepts and its evaluation , 2003, 33rd Annual Frontiers in Education, 2003. FIE 2003..

[25]  Ehud Reiter,et al.  Book Reviews: Building Natural Language Generation Systems , 2000, CL.

[26]  Michael Gertz,et al.  A Model and Framework for Visualization Exploration , 2007, IEEE Transactions on Visualization and Computer Graphics.

[27]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.