Assessing the Usability of Video Browsing and Summarization Techniques

Since 1994, the Informedia group at Carnegie Mellon University has been developing and evaluating surrogates, summary interfaces, and visualizations for accessing digital video collections containing thousands of documents, millions of shots, and terabytes of data. This chapter samples Informedia user studies that have taken place through the years, reporting on how these studies provide a user pull complementing the technology push as automated video processing advances. Specifically, this chapter examines indicative video summaries— i.e., the assessment of video surrogates meant to help users better judge the relevance of the source program for their task at hand. The merits of discount usability techniques for iterative improvement and evaluation are presented, as well as the structure of formal empirical investigations with end users that have ecological validity while addressing the human computer interaction metrics of efficiency, effectiveness, and satisfaction. Lessons learned from such studies are reported with respect to video summarization and browsing, ranging from the simplest portrayal of a single thumbnail to represent video stories, to collections of thumbnails in storyboards, to playable video skims, and to video collages with multiple synchronized information perspectives. Advances in interactive video retrieval are charted through the annual National Institiute for Standards Technology (NIST) TRECVID evaluation forum, concluding with discussion on difficulties in evaluating video summarization and browsing interfaces.

[1]  Marti A. Hearst,et al.  Finding the flow in web site search , 2002, CACM.

[2]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[3]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[4]  Paul Over,et al.  The trecvid 2007 BBC rushes summarization evaluation pilot , 2007, TVS '07.

[5]  Ben Shneiderman,et al.  Visual information seeking: tight coupling of dynamic query filters with starfield displays , 1994, CHI '94.

[6]  Michael G. Christel,et al.  The effect of text in storyboards for video navigation , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[7]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[8]  Alexander G. Hauptmann.  Lessons for the Future from a Decade of Informedia Video Analysis Research , 2005, CIVR.

[9]  Peter G. B. Enser,et al.  Retrieval of Archival Moving Imagery - CBIR Outside the Frame? , 2002, CIVR.

[10]  Alfred Kobsa,et al.  An empirical comparison of three commercial information visualization systems , 2001, IEEE Symposium on Information Visualization, 2001. INFOVIS 2001..

[11]  Janni Nielsen,et al.  Getting access to what goes on in people's heads?: reflections on the think-aloud technique , 2002, NordiCHI '02.

[12]  Boon-Lock Yeo,et al.  Retrieving and visualizing video , 1997, CACM.

[13]  Wei-Hao Lin,et al.  Clever clustering vs. simple speed-up for summarizing rushes , 2007, TVS '07.

[14]  Ramesh C. Jain,et al.  ACM SIGMM retreat report on future directions in multimedia research , 2005, TOMCCAP.

[15]  Michael G. Christel,et al.  Evolving video skims into useful multimedia abstractions , 1998, CHI.

[16]  Michael G. Christel.  Examining user interactions with video retrieval systems , 2007, Electronic Imaging.

[17]  Behzad Shahraray,et al.  On the applications of multimedia processing to communications , 1998, Proc. IEEE.

[18]  Michael G. Christel,et al.  Addressing the challenge of visual information access from digital image and video libraries , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[19]  John R. Smith,et al.  Large-scale concept ontology for multimedia , 2006, IEEE MultiMedia.

[20]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[21]  A. Murat Tekalp,et al.  Two-stage hierarchical video summary extraction to match low-level user browsing preferences , 2003, IEEE Trans. Multim..

[22]  Michael G. Christel,et al.  Improving Access to a Digital Video Library , 1997, INTERACT.

[23]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[24]  Alan F. Smeaton,et al.  Designing the User Interface for the Físchlár Digital Video Library , 2006, J. Digit. Inf..

[25]  Tobun Dorbin Ng,et al.  Collages as dynamic summaries for news video , 2002, MULTIMEDIA '02.

[26]  Michael G. Christel.  Establishing the utility of non-text search for news video retrieval with real world users , 2007, ACM Multimedia.

[27]  A. Murat Tekalp,et al.  Automatic soccer video analysis and summarization , 2003, IEEE Trans. Image Process..

[28]  Michael G. Christel,et al.  Mining Novice User Activity with TRECVID Interactive Retrieval Tasks , 2006, CIVR.

[29]  Gary Marchionini,et al.  Multimodal surrogates for video browsing , 1999, DL '99.

[30]  Michael G. Christel,et al.  Information Visualization Within a Digital Video Library , 1998, Journal of Intelligent Information Systems.

[31]  Zygmunt Pizlo,et al.  Automated video program summarization using speech transcripts , 2006, IEEE Transactions on Multimedia.

[32]  Simon King,et al.  From context to content: leveraging context to infer media metadata , 2004, MULTIMEDIA '04.

[33]  Yihong Gong,et al.  Lessons Learned from Building a Terabyte Digital Video Library , 1999, Computer.

[34]  Michael G. Christel.  Evaluation and user studies with respect to video summarization and browsing , 2006, Electronic Imaging.

[35]  Gary Marchionini,et al.  The relative effectiveness of concept-based versus content-based video retrieval , 2004, MULTIMEDIA '04.

[36]  Marcel Worring,et al.  Assessing User Behaviour in News Video Retrieval , 2005 .

[37]  Shingo Uchihashi,et al.  An interactive comic book presentation for exploring video , 2000, CHI.

[38]  Gary Marchionini,et al.  Key frame preview techniques for video browsing , 1998, DL '98.

[39]  Marcel Worring,et al.  Mediamill: Advanced Browsing in News Video Archives , 2006, CIVR.

[40]  Joseph F. Dumas,et al.  A Practical Guide to Usability Testing , 1993 .

[41]  Qingming Huang,et al.  Subjective evaluation criterion for selecting affective features and modeling highlights , 2006, Electronic Imaging.

[42]  John Adcock,et al.  Interactive search in large video collections , 2005, CHI EA '05.

[43]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[44]  Stephen W. Smoliar,et al.  Video parsing and browsing using compressed data , 1995, Multimedia Tools and Applications.

[45]  Sara Shatford,et al.  Analyzing the Subject of a Picture: A Theoretical Approach , 1986 .

[46]  Dragutin Petkovic,et al.  Key to effective video retrieval: effective cataloging and browsing , 1998, MULTIMEDIA '98.

[47]  Ba Tu Truong,et al.  Video abstraction: A systematic review and classification , 2007, TOMCCAP.

[48]  Stefan M. Rüger,et al.  Three Interfaces for Content-Based Access to Image Collections , 2004, CIVR.

[49]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[50]  Jonathan J. Hull,et al.  Refocusing Multimedia Research on Short Clips , 2005, IEEE Multim..

[51]  Alexander G. Hauptmann,et al.  Successful approaches in the TREC video retrieval evaluations , 2004, MULTIMEDIA '04.

[52]  Jakob Nielsen,et al.  Evaluating the thinking-aloud technique for use by computer scientists , 1993 .

[53]  Wolfgang Effelsberg,et al.  Video abstracting , 1997, CACM.

[54]  Marcel Worring,et al.  A Learned Lexicon-Driven Paradigm for Interactive Video Retrieval , 2007, IEEE Transactions on Multimedia.

[55]  Michael G. Christel,et al.  Finding the right shots: assessing usability and performance of a digital video library interface , 2004, MULTIMEDIA '04.