Advancing User-Centered Evaluation of Visual Analytic Environments through Contests

In this paper, the authors describe the Visual Analytics Science and Technology (VAST) Symposium contests run in 2006 and 2007 and the VAST 2008 and 2009 challenges. These contests were designed to provide researchers with a better understanding of the tasks and data that face potential end users. Access to these end users is limited because of time constraints and the classified nature of the tasks and data. In that respect, the contests serve as an intermediary, with the metrics and feedback serving as measures of utility to the end users. The authors summarize the lessons learned and the future directions for VAST Challenges.

[1]  Jean Scholtz,et al.  Evaluating Visual Analytics: The 2007 Visual Analytics Science and Technology Symposium Contest | NIST , 2008 .

[2]  Daniel A. Keim,et al.  Visual Analytics: Definition, Process, and Challenges , 2008, Information Visualization.

[3]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[4]  Jean Scholtz,et al.  VAST 2007 Contest - Blue Iguanodon , 2007, 2007 IEEE Symposium on Visual Analytics Science and Technology.

[5]  Mark A. Whiting,et al.  Creating realistic, scenario-based synthetic data for test and evaluation of information analytics software , 2008, BELIV.

[6]  Ellen M. Voorhees,et al.  TREC: Continuing information retrieval's tradition of experimentation , 2007, CACM.

[7]  Jean Scholtz,et al.  VAST 2008 Challenge: Introducing mini-challenges , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[8]  Jean-Daniel Fekete,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS , 2022 .

[9]  Janice Ginny Redish,et al.  Expanding usability testing to evaluate complex systems , 2007 .

[10]  Jean Scholtz,et al.  Evaluating Visual Analytics at the 2007 VAST Symposium Contest , 2008, IEEE Computer Graphics and Applications.

[11]  Jean Scholtz,et al.  VAST 2006 Contest - A tale of Alderwood | NIST , 2006 .

[12]  Jean Scholtz,et al.  VAST 2006 Contest - A Tale of Alderwood , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.