Measuring user performance during interactions with digital video collections

With more and more digital videos found online, video retrieval researchers have begun to create various representations or surrogates for digital videos, such as poster frames, storyboards, video skims and fast forwards. How to evaluate the effectiveness of these video surrogates has become an issue for researchers. This paper proposes two general classes of user tasks—recognition tasks and tasks requiring inference—for which performance measures were developed. The measures include graphical object recognition, textual object recognition, action recognition, free-text gist determination, multiple-choice gist determination and visual gist determination. The preliminary results from two user studies applying these six measures are also discussed in this paper.

[1]  Frank Nack,et al.  Hybrid narrative and categorical strategies for interactive and dynamic video presentation generation , 2000, New Rev. Hypermedia Multim..

[2]  Anoop Gupta,et al.  Auto-summarization of audio-video presentations , 1999, MULTIMEDIA '99.

[3]  R. Shepard.  Recognition memory for words, sentences, and pictures , 1967 .

[4]  David Bordwell,et al.  Film Art: An Introduction , 1979 .

[5]  Michael G. Christel,et al.  Improving Access to a Digital Video Library , 1997, INTERACT.

[6]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[7]  Gary Marchionini,et al.  Key frame preview techniques for video browsing , 1998, DL '98.

[8]  W. Grabe.  CURRENT DEVELOPMENTS IN SECOND LANGUAGE READING RESEARCH , 1991 .

[9]  Gary Marchionini,et al.  Dynamic key frame presentation techniques for augmenting video browsing , 1998, AVI '98.

[10]  Gary Marchionini,et al.  Comprehension and Object Recognition Capabilities for Presentations of Simultaneous Video Key Frame Surrogates , 1997, ECDL.

[11]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[12]  Edward Lee Elliott.  Watch-grab-arrange-see : thinking with motion images via streams and collages , 1993 .

[13]  Abby Goodrum.  Evaluation of Text-Based and Image-Based Representations for Moving Image Documents , 1997 .

[14]  Gary Marchionini,et al.  How fast is too fast? evaluating fast forward surrogates for digital video , 2003, 2003 Joint Conference on Digital Libraries, 2003. Proceedings..

[15]  Erwin Panofsky,et al.  Meaning in the Visual Arts: Papers in and on Art History , 1955 .

[16]  Abby Goodrum,et al.  Multidimensional scaling of video surrogates , 2001, J. Assoc. Inf. Sci. Technol..

[17]  W. Kintsch,et al.  Strategies of discourse comprehension , 1983 .

[18]  Steven M. Drucker,et al.  SmartSkip: consumer level browsing and skipping of digital video content , 2002, CHI.

[19]  Allan H. Gilbert,et al.  Studies In Iconology: Humanistic Themes In The Art Of The Renaissance , 1939 .

[20]  Brian C. O'Connor,et al.  Modelling what users see when they look at images: a cognitive viewpoint , 2002, J. Documentation.

[21]  Gary Marchionini,et al.  Performance of Visual, Verbal, and Combined Video Surrogates. , 1999 .

[22]  Michael G. Christel,et al.  Evolving video skims into useful multimedia abstractions , 1998, CHI.

[23]  T. Grodal.  Moving Pictures: A New Theory of Film Genres, Feelings, and Cognition , 1997 .

[24]  Gary Marchionini,et al.  The Open Video Digital Library , 2002, D Lib Mag..

[25]  Gary Marchionini,et al.  Alternative Surrogates for Video Objects in a Digital Library: Users' Perspectives on Their Relative Usability , 2002, ECDL.