Evaluating information visualization applications with focus groups: the CourseVis experience

This paper reports our experience of evaluating an application that uses visualization approaches to support instructors in Web based distance education. The evaluation took place in three stages: a focus group, an experimental study, and a semi-structured interview. In this paper we focus our attention on the focus group, and we will show how this evaluation approach can be very effective in uncovering unexpected problems that cannot be identified with analytic evaluations or controlled experiments.

[1]  Giacomo Mauro DAriano.  The Handbook for Focus Group Research. , 1994 .

[2]  Peter Johnson,et al.  Human computer interaction: Psychology, task analysis, and software engineering , 1992 .

[3]  Richard A. Krueger,et al.  Focus groups : a practical guide for applied research / by Richard A. Krueger , 1989 .

[4]  Thomas L. Greenbaum.  handbook for focus group research , 1993 .

[5]  Chaomei Chen,et al.  Top 10 Unsolved Information Visualization Problems , 2005, IEEE Computer Graphics and Applications.

[6]  Gregory D. Abowd,et al.  Human-Computer Interaction. Second Edition. , 1998 .

[7]  Giuseppe Santucci,et al.  BELIV'08: Beyond time and errors: novel evaluation methods for information visualization , 2008, CHI Extended Abstracts.

[8]  Janet Mancini Billson,et al.  Focus Groups: A Practical Guide for Applied Research , 1989 .

[9]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[10]  Riccardo Mazza,et al.  Exploring Usage Analysis in Learning Systems: Gaining Insights From Visualisations , 2005 .

[11]  Vania Dimitrova,et al.  Visualising student tracking data to support instructors in web-based distance education , 2004, WWW Alt. '04.

[12]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[13]  Jakob Nielsen,et al.  Interface: The Use and Misuse of Focus Groups , 1997, IEEE Softw..

[14]  Bonnie Kaplan,et al.  Qualitative Research Methods for Evaluating Computer Information Systems , 2005 .

[15]  Melanie Tory,et al.  Human factors in visualization research , 2004, IEEE Transactions on Visualization and Computer Graphics.

[16]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[17]  Paris Avgeriou,et al.  Towards a Pattern Language for Learning Management Systems , 2003, J. Educ. Technol. Soc..

[18]  Vania Dimitrova,et al.  Generation of graphical representations of student tracking data in course management systems , 2005, Ninth International Conference on Information Visualisation (IV'05).

[19]  Riccardo Mazza,et al.  Using information visualisation to facilitate instructors in web-based distance learning , 2004 .