Human performance measures for video retrieval

In this paper, we describe the challenges of assessing human performance during video retrieval episodes and describe several measures of human performance that have been used in developing visual surrogates for the Open Video Digital Library (http://www.open-video.org). These include two sets of cognitive performance measures that aim to assess human recognition and inference and a set of attitudinal measures that aim to assess user satisfaction with video surrogates.

[1]  Thijs Westerveld,et al.  Benchmarking multimedia search in structured collections , 2006, MIR '06.

[2]  Gary Marchionini,et al.  Measures of User Performance in Video Retrieval Research , 2003 .

[3]  Gary Marchionini,et al.  The Open Video Digital Library: A Möbius strip of research and practice , 2006, J. Assoc. Inf. Sci. Technol..

[4]  Gary Marchionini,et al.  Examining the effectiveness of real-time query expansion , 2007, Inf. Process. Manag..

[5]  Gary Marchionini,et al.  The Open Video Digital Library: A Möbius strip of research and practice , 2006 .

[6]  Gary Marchionini,et al.  Surrogation for Digital Video : A Design Framework , 2006 .

[7]  Ying Li,et al.  Creating MAGIC: system for generating learning object metadata for instructional content , 2005, MULTIMEDIA '05.

[8]  Bernd Fröhlich,et al.  The cubic mouse: a new device for three-dimensional input , 2000, CHI.

[9]  Gary Marchionini,et al.  Measuring user performance during interactions with digital video collections , 2005, ASIST.

[10]  Spencer W. Thomas.  Lessons learned from building a terabyte digital video library , 1999 .

[11]  Christian Fluhr,et al.  Usage-oriented multimedia information retrieval technological evaluation , 2006, MIR '06.

[12]  Gary B. Reid,et al.  Critical SWAT Values for Predicting Operator Overload , 1988 .

[13]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[14]  Gary Marchionini,et al.  Text or Pictures? An Eyetracking Study of How People View Digital Video Surrogates , 2003, CIVR.

[15]  Wessel Kraaij,et al.  TRECVID 2005-An Introduction , 2005 .

[16]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[17]  Yihong Gong,et al.  Lessons Learned from Building a Terabyte Digital Video Library , 1999, Computer.

[18]  Gary Marchionini,et al.  Smartlinks in a video-based collaborative distance learning system: a cognitive model and evaluation study , 2004 .

[19]  Karen Spärck Jones.  What's the value of TREC: is there a gap to jump or a chasm to bridge? , 2006, SIGF.

[20]  Gary Marchionini,et al.  Report on ACM SIGIR 2006 workshop on evaluating exploratory search systems , 2006, SIGF.

[21]  Peter Bruza,et al.  Interactive Internet search: keyword, directory and query reformulation mechanisms compared , 2000, SIGIR '00.

[22]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.