The scholarly impact of TRECVid (2003-2009)

This paper reports on an investigation into the scholarly impact of the TRECVid (Text Retrieval and Evaluation Conference, Video Retrieval Evaluation) benchmarking conferences between 2003 and 2009. The contribution of TRECVid to research in video retrieval is assessed by analyzing publication content to show the development of techniques and approaches over time and by analyzing publication impact through publication numbers and citation analysis. Popular conference and journal venues for TRECVid publications are identified in terms of number of citations received. For a selection of participants at different career stages, the relative importance of TRECVid publications in terms of citations vis a vis their other publications is investigated. TRECVid, as an evaluation conference, provides data on which research teams ‘scored’ highly against the evaluation criteria and the relationship between ‘top scoring’ teams at TRECVid and the ‘top scoring’ papers in terms of citations is analyzed. A strong relationship was found between ‘success’ at TRECVid and ‘success’ at citations both for high scoring and low scoring teams. The implications of the study in terms of the value of TRECVid as a research activity, and the value of bibliometric analysis as a research evaluation tool, are discussed. © 2011 Wiley Periodicals, Inc.

[1]  Karl R. Popper.  The Logic of Scientific Discovery. , 1977 .

[2]  Stephen E. Robertson,et al.  On the history of evaluation in IR , 2008, J. Inf. Sci..

[3]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[4]  Peter G. B. Enser,et al.  Activity theory and information seeking , 2008 .

[5]  Derek de Solla Price,et al.  A general theory of bibliometric and other cumulative advantage processes , 1976, J. Am. Soc. Inf. Sci..

[6]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[7]  Dag W. Aksnes,et al.  Citation rates and perceptions of scientific contribution , 2006 .

[8]  Cassidy R. Sugimoto,et al.  Visualizing changes over time: A history of information retrieval through the lens of descriptor tri-occurrence mapping , 2010, J. Inf. Sci..

[9]  Padraig Cunningham,et al.  Relative status of journal and conference publications in computer science , 2010, Commun. ACM.

[10]  Peter G. B. Enser,et al.  The evolution of visual information retrieval , 2008, J. Inf. Sci..

[11]  Henk F. Moed,et al.  Developing Bibliometric Indicators of Research Performance in Computer Science , 2007 .

[12]  Jacques Wainer,et al.  Patterns of bibliographic references in the ACM published papers , 2011, Inf. Process. Manag..

[13]  Albert N. Link,et al.  Economic impact assessment of NIST's text REtrieval conference (TREC) program. Final report , 2010 .

[14]  Peter G. B. Enser,et al.  Visual image retrieval , 2008, Annu. Rev. Inf. Sci. Technol..

[15]  Judit Bar-Ilan,et al.  Which h-index? — A comparison of WoS, Scopus and Google Scholar , 2008, Scientometrics.

[16]  C. J. van Rijsbergen,et al.  The geometry of information retrieval , 2004 .

[17]  Niels Ole Pors,et al.  Information retrieval, experimental models and statistical analysis , 2000, J. Documentation.