This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS

Information visualization (InfoVis) is now an accepted and growing field, but questions remain about the best uses for and the maturity of novel visualizations. Usability studies and controlled experiments are helpful, but generalization is difficult. We believe that the systematic development of benchmarks will facilitate the comparison of techniques and help identify their strengths under different conditions. We were involved in the organization and management of three InfoVis contests for the 2003, 2004, and 2005 IEEE InfoVis Symposia, which requested teams to report on insights gained while exploring data. We give a summary of the state of the art of evaluation in InfoVis, describe the three contests, summarize their results, discuss outcomes and lessons learned, and conjecture the future of visualization contests. All materials produced by the contests are archived in the InfoVis benchmark repository.

[1]  Diansheng Guo,et al.  Visual Inquiry Toolkit - An Integrated Approach for Exploring and Interpreting Space-Time, Multivariate Patterns. , 2006, AutoCarto research symposium.

[2]  A. MacEachren,et al.  Space-Time-Attribute Analysis and Visualization of U . S . Company Data , 2005 .

[3]  Ning Lu,et al.  Designing a Collaborative Visual Analytics Tool for Social and Technological Change Prediction , 2009, IEEE Computer Graphics and Applications.

[4]  Catherine Plaisant,et al.  SpaceTree: supporting exploration in large node link tree, design evolution and empirical evaluation , 2002, IEEE Symposium on Information Visualization, 2002. INFOVIS 2002..

[5]  N. F. F. Ebecken,et al.  Data mining II , 2000 .

[6]  Le Song,et al.  WilmaScope Graph Visualisation , 2004 .

[7]  Daniel A. Keim,et al.  Test Data Sets for Evaluating Data Visualization Techniques , 1995, Perceptual Issues in Visualization.

[8]  Haim Levkowitz,et al.  Perceptual Issues in Visualization , 1995, IFIP Series on Computer Graphics.

[9]  Jean-Philippe Domenger,et al.  EVAT: Environment for Visualization and Analysis of Trees , 2003 .

[10]  Jin Young Hong,et al.  Zoomology: ComparingTwo Large Hierarchical Trees , 2003 .

[11]  Michael Spenke,et al.  InfoZoom − Analysing Formula One racing results with an interactive data mining and visualisation tool , 2000 .

[12]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[13]  Weimao Ke,et al.  Studying the emerging global brain: Analyzing and visualizing the impact of co-authorship teams , 2005, Complex..

[14]  Mary Czerwinski,et al.  Empirical evaluation of information visualizations: an introduction , 2000, Int. J. Hum. Comput. Stud..

[15]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[16]  Serdar Tasiran,et al.  TreeJuxtaposer: scalable tree comparison using Focus+Context with guaranteed visibility , 2003, ACM Trans. Graph..

[17]  J. Gregory Trafton,et al.  Turning pictures into numbers: extracting and generating information from complex visualizations , 2000, Int. J. Hum. Comput. Stud..

[18]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[19]  Jock D. Mackinlay,et al.  Automating the design of graphical presentations of relational information , 1986, TOGS.

[20]  Mark A. Whiting,et al.  Threat stream data generator: creating the known unknowns for test and evaluation of visual analytics tools , 2006, BELIV '06.

[21]  Antony Unwin,et al.  Graphics of a Large Dataset , 2006 .

[22]  Jon M. Kleinberg,et al.  Overview of the 2003 KDD Cup , 2003, SKDD.

[23]  Jean Scholtz,et al.  User-centered Evaluations of Ubicomp Applications , 2002 .

[24]  Christopher Fry,et al.  On your marks, get set, browse! , 1997, CHI Extended Abstracts.

[25]  Jonathan G. Fiscus,et al.  Measurements in support of research accomplishments , 2000, CACM.

[26]  Alfred Kobsa,et al.  Benefits of information visualization systems for administrative data analysts , 2003, Proceedings on Seventh International Conference on Information Visualization, 2003. IV 2003..

[27]  Chris North,et al.  An Evaluation of Microarray Visualization Tools for Biological Insight , 2004, IEEE Symposium on Information Visualization.

[28]  Lynette Hirschman,et al.  Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3) , 1993, CL.

[29]  Jason Baumgartner,et al.  Treemap, Radial Tree, and 3D Tree Visualizations , 2003 .

[30]  Jean Scholtz,et al.  A Framework for Evaluating Collaborative Systems in the Real World , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[31]  Jan W. Buzydlowski,et al.  An Associative Information Visualizer , 2004 .

[32]  Jean-Philippe Domenger,et al.  Exploring InfoVis Publication History with Tulip , 2004 .

[33]  Daniel A. Keim,et al.  Exploring and Visualizing the History of InfoVis , 2004 .

[34]  Giuseppe Santucci,et al.  BELIV'08: Beyond time and errors: novel evaluation methods for information visualization , 2008, CHI Extended Abstracts.

[35]  Jean Scholtz,et al.  VAST 2006 Contest - A Tale of Alderwood , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[36]  Chenchaomei,et al.  Empirical evaluation of information visualizations , 2000 .

[37]  Alan M. MacEachren,et al.  A Visual Inquiry System for Space-Time and Multivariate Patterns ( VIS-STAMP ) , 2005 .

[38]  Chris North,et al.  Toward measuring visualization insight , 2006, IEEE Computer Graphics and Applications.

[39]  Jean-Daniel Fekete,et al.  The InfoVis Toolkit , 2004, IEEE Symposium on Information Visualization.

[40]  Tzu-Wei Hsu,et al.  MonkEllipse: Visualizing the History of Information Visualization , 2004 .

[41]  D. Morse,et al.  Comparison of multiple taxonomic hierarchies using TaxoNote , 2003 .

[42]  Diansheng Guo,et al.  Supporting the Process of Exploring and Interpreting Space–Time Multivariate Patterns: The Visual Inquiry Toolkit , 2008, Cartography and geographic information science.

[43]  Mary Czerwinski,et al.  Understanding Eight Years of InfoVis Conferences Using PaperLens , 2004 .

[44]  Jan Schröder,et al.  CPS - Company Positioning System , 2005 .

[45]  Rosane Minghim,et al.  Content-based text mapping using multi-dimensional projections for exploration of document collections , 2006, Electronic Imaging.

[46]  Jan W. Buzydlowski,et al.  An Associative Information Visualizer , 2004, IEEE Symposium on Information Visualization.

[47]  Jean Scholtz,et al.  Glass Box: An Instrumented Infrastructure for Supporting Human Interaction with Information , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[48]  Donna K. Harman,et al.  Overview of the Sixth Text REtrieval Conference (TREC-6) , 1997, Inf. Process. Manag..

[49]  Jin Chen,et al.  A Visualization System for Space-Time and Multivariate Patterns (VIS-STAMP) , 2006, IEEE Transactions on Visualization and Computer Graphics.

[50]  Philip S. Yu,et al.  Epi-SPIRE: a system for environmental and public health activity monitoring , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).