Controlled user evaluations of information visualization interfaces for text retrieval: Literature review and meta-analysis

This review describes experimental designs (users, search tasks, measures, etc.) used by 31 controlled user studies of information visualization (IV) tools for textual information retrieval (IR) and a meta-analysis of the reported statistical effects. Comparable experimental designs allow research designers to compare their results with other reports, and support the development of experimentally verified design guidelines concerning which IV techniques are better suited to which types of IR tasks. The studies generally use a within-subject design with 15 or more undergraduate students performing browsing to known-item tasks on sets of at least 1,000 full-text articles or Web pages on topics of general interest-news. Results of the meta-analysis (N = 8) showed no significant effects of the IV tool as compared with a text-only equivalent, but the set shows great variability suggesting an inadequate basis of comparison. Experimental design recommendations are provided which would support comparison of existing IV tools for IR usability testing. © 2008 Wiley Periodicals, Inc.

[1]  Jakob Nielsen,et al.  User Empowerment and the Fun Factor - Questions and Answers with Jakob Nielsen , 2005, Funology.

[2]  Susan T. Dumais,et al.  A comparison of symbolic and spatial filing , 1985, CHI '85.

[3]  Nicholas J. Belkin,et al.  Evaluation of a tool for visualization of information retrieval results , 1996, SIGIR '96.

[4]  Alfred Kobsa.  User Experiments with Tree Visualization Systems , 2004 .

[5]  Donald Byrd,et al.  A scrollbar-based visualization for document navigation , 1999, DL '99.

[6]  Sherry Koshman.  Testing user interaction with a prototype visualization-based information retrieval system , 2005, J. Assoc. Inf. Sci. Technol..

[7]  Mary Czerwinski,et al.  An initial examination of ease of use for 2D and 3D information visualizations of web content , 2000, Int. J. Hum. Comput. Stud..

[8]  ShneidermanBen.  Designing for fun , 2004 .

[9]  Jay F. Nunamaker,et al.  A Visual Framework for Knowledge Discovery on the Web: An Empirical Study of Business Intelligence Exploration , 2005, J. Manag. Inf. Syst..

[10]  Janice Singer,et al.  Studying Software Engineers: Data Collection Techniques for Software Field Studies , 2005, Empirical Software Engineering.

[11]  D.A. Carr,et al.  Browsing thumbnails: a comparison of three techniques , 2004, 26th International Conference on Information Technology Interfaces, 2004..

[12]  Dennis Wixon,et al.  User performance with command, menu, and iconic interfaces , 1985, CHI '85.

[13]  Ben Shneiderman,et al.  The Craft of Information Visualization: Readings and Reflections , 2003 .

[14]  J. H. Erik Andriessen.  The Why, How and What to Evaluate of Interaction Technology: A Review and Proposed Integration , 1996 .

[15]  Alfred Kobsa,et al.  An empirical comparison of three commercial information visualization systems , 2001, IEEE Symposium on Information Visualization, 2001. INFOVIS 2001..

[16]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[17]  Harald Reiterer,et al.  MedioVis - A User-Centred Library Metadata Browser , 2005, ECDL.

[18]  Mary Czerwinski,et al.  Data mountain: using spatial memory for document management , 1998, UIST '98.

[19]  Marc M. Sebrechts,et al.  Visualization of search results: a comparative evaluation of text, 2D, and 3D interfaces , 1999, SIGIR '99.

[20]  Alistair G. Sutcliffe,et al.  Evaluating the effectiveness of visual user interfaces for information retrieval , 2000, Int. J. Hum. Comput. Stud..

[21]  Padraic Neville,et al.  A comparison of 2-D visualizations of hierarchies , 2001, IEEE Symposium on Information Visualization, 2001. INFOVIS 2001..

[22]  D. Norman.  Emotional design : why we love (or hate) everyday things , 2004 .

[23]  Mikael B. Skov,et al.  A review of research methods in children's technology design , 2005, IDC '05.

[24]  Timothy Cribbin,et al.  Mapping semantic information in virtual space: dimensions, variance and individual differences , 2000, Int. J. Hum. Comput. Stud..

[25]  James Allan,et al.  Aspect windows, 3-D visualizations, and indirect comparisons of information retrieval systems , 1998, SIGIR '98.

[26]  Chaomei Chen,et al.  Empirical studies of information visualization: a meta-analysis , 2000, Int. J. Hum. Comput. Stud..

[27]  Timothy Cribbin,et al.  Visual-spatial exploration of thematic spaces: a comparative study of three visualization models , 2001, IS&T/SPIE Electronic Imaging.

[28]  John A. Waterworth,et al.  Effects of desktop 3D world design on user navigation and search performance , 2000, 2000 IEEE Conference on Information Visualization. An International Conference on Computer Visualization and Graphics.

[29]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[30]  Timothy Cribbin,et al.  Browsing a document collection represented in two- and three-dimensional virtual information space , 2005, Int. J. Hum. Comput. Stud..

[31]  Carlos Ferreira,et al.  Visual application for the analysis of Web-based information systems usage: a preliminary usability evaluation , 2004 .

[32]  D. Levine,et al.  A neuropsychological theory of positive affect and its influence on cognition. , 1999, Psychological review.

[33]  Ben Shneiderman,et al.  Designing for fun: how can we design user interfaces to be more fun? , 2004, INTR.

[34]  Hsinchun Chen,et al.  Internet Browsing and Searching: User Evaluations of Category Map and Concept Space Techniques , 1998, J. Am. Soc. Inf. Sci..

[35]  Hsinchun Chen,et al.  Internet browsing: visualizing category map by fisheye and fractal views , 2002, Proceedings. International Conference on Information Technology: Coding and Computing.

[36]  Gary Marchionini,et al.  Evaluation and evolution of a browse and search interface: relation browser , 2005, DG.O.

[37]  Ramesh Sharda,et al.  Clustering-Based Visual Interfaces for Presentation of Web Search Results: An Empirical Investigation , 2005, Inf. Syst. Frontiers.

[38]  Stefano Mizzaro,et al.  Evaluating user interfaces to information retrieval systems: a case study on user support , 1996, SIGIR '96.

[39]  A. Sengupta,et al.  Query by templates: using the shape of information to search next-generation databases , 2006, IEEE Transactions on Professional Communication.

[40]  Maximilian Eibl,et al.  DEViD: A Media Design and Software Ergonomics Integrating Visualization for Document Retrieval , 2002, Inf. Vis..

[41]  David A. Carr.  Guidelines for designing information visualization applications , 1999 .

[42]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[43]  Nicholas J. Belkin,et al.  The TREC Interactive Tracks: Putting the User into Search , 2005 .

[44]  Ryen W. White,et al.  Exploratory Search , 2008 .

[45]  Mark H. Chignell,et al.  Individual Differences in Exploration Using Desktop VR , 2003, J. Assoc. Inf. Sci. Technol..

[46]  Gregory B. Newby,et al.  Empirical Study of a 3D Visualization for Information Retrieval Tasks , 2004, Journal of Intelligent Information Systems.

[47]  Xia Lin.  Map displays for information retrieval , 1997 .

[48]  Aravindan Veerasamy,et al.  Effectiveness of a graphical display of retrieval results , 1997, SIGIR '97.

[49]  Julian Newman,et al.  Validation, evaluation and virtual enterprise research , 2001, Proceedings Tenth IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. WET ICE 2001.

[50]  Concepción Pérez,et al.  3D visualization of text collections: an experimental study to assess the usefulness of 3D , 2004 .

[51]  Robert H. Halstead,et al.  Parallel Symbolic Computing , 1986, Computer.

[52]  Andrian Marcus,et al.  Supporting the evolution of a software visualization tool through usability studies , 2005, 13th International Workshop on Program Comprehension (IWPC'05).