Evaluation and user studies with respect to video summarization and browsing

The Informedia group at Carnegie Mellon University has since 1994 been developing and evaluating surrogates, summary interfaces, and visualizations for accessing digital video collections containing thousands of documents, millions of shots, and terabytes of data. This paper surveys the Informedia user studies that have taken place through the years, reporting on how these studies can provide a user pull complementing the technology push as automated video processing advances. The merits of discount usability techniques for iterative improvement and evaluation are presented, as well as the structure of formal empirical investigations with end users that have ecological validity while addressing the human computer interaction metrics of efficiency, effectiveness, and satisfaction. The difficulties in evaluating video summarization and browsing interfaces are discussed. Lessons learned from Informedia user studies are reported with respect to video summarization and browsing, ranging from the simplest portrayal of a single thumbnail to represent video stories, to collections of thumbnails in storyboards, to playable video skims, to video collages with multiple synchronized information perspectives.

[1]  Sara Shatford,et al.  Analyzing the Subject of a Picture: A Theoretical Approach , 1986 .

[2]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[3]  Jakob Nielsen,et al.  Evaluating the thinking-aloud technique for use by computer scientists , 1993 .

[4]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[5]  Ben Shneiderman,et al.  Visual information seeking: tight coupling of dynamic query filters with starfield displays , 1994, CHI '94.

[6]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[7]  Wolfgang Effelsberg,et al.  Video abstracting , 1997, CACM.

[8]  Michael G. Christel,et al.  Improving Access to a Digital Video Library , 1997, INTERACT.

[9]  Boon-Lock Yeo,et al.  Retrieving and visualizing video , 1997, CACM.

[10]  Michael G. Christel,et al.  Evolving video skims into useful multimedia abstractions , 1998, CHI.

[11]  Dragutin Petkovic,et al.  Key to effective video retrieval: effective cataloging and browsing , 1998, MULTIMEDIA '98.

[12]  Behzad Shahraray,et al.  On the applications of multimedia processing to communications , 1998, Proc. IEEE.

[13]  Yihong Gong,et al.  Lessons Learned from Building a Terabyte Digital Video Library , 1999, Computer.

[14]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[15]  Gary Marchionini,et al.  Multimodal surrogates for video browsing , 1999, DL '99.

[16]  Shingo Uchihashi,et al.  An interactive comic book presentation for exploring video , 2000, CHI.

[17]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[18]  Anoop Gupta,et al.  Browsing digital video , 2000, CHI.

[19]  Alfred Kobsa,et al.  An empirical comparison of three commercial information visualization systems , 2001, IEEE Symposium on Information Visualization, 2001. INFOVIS 2001..

[20]  Michael G. Christel,et al.  The effect of text in storyboards for video navigation , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[21]  Alan F. Smeaton,et al.  Designing the User Interface for the Físchlár Digital Video Library , 2006, J. Digit. Inf..

[22]  Michael G. Christel.  Accessing News Video Libraries through Dynamic Information Extraction, Summarization, and Visualization , 2002, Visual Interfaces to Digital Libraries.

[23]  Tobun Dorbin Ng,et al.  Collages as dynamic summaries for news video , 2002, MULTIMEDIA '02.

[24]  Janni Nielsen,et al.  Getting access to what goes on in people's heads?: reflections on the think-aloud technique , 2002, NordiCHI '02.

[25]  Marti A. Hearst,et al.  Finding the flow in web site search , 2002, CACM.

[26]  Peter G. B. Enser,et al.  Retrieval of Archival Moving Imagery - CBIR Outside the Frame? , 2002, CIVR.

[27]  Alexander G. Hauptmann,et al.  Successful approaches in the TREC video retrieval evaluations , 2004, MULTIMEDIA '04.

[28]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[29]  Michael G. Christel,et al.  Finding the right shots: assessing usability and performance of a digital video library interface , 2004, MULTIMEDIA '04.

[30]  Michael G. Christel,et al.  Information Visualization Within a Digital Video Library , 1998, Journal of Intelligent Information Systems.

[31]  Simon King,et al.  From context to content: leveraging context to infer media metadata , 2004, MULTIMEDIA '04.

[32]  Michael G. Christel,et al.  Addressing the challenge of visual information access from digital image and video libraries , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[33]  Jonathan J. Hull,et al.  Refocusing Multimedia Research on Short Clips , 2005, IEEE Multim..

[34]  Stephen W. Smoliar,et al.  Video parsing and browsing using compressed data , 1995, Multimedia Tools and Applications.

[35]  Alexander G. Hauptmann,et al.  The Use and Utility of High-Level Semantic Features in Video Retrieval , 2005, CIVR.

[36]  Ramesh C. Jain,et al.  ACM SIGMM retreat report on future directions in multimedia research , 2005, TOMCCAP.

[37]  Alexander G. Hauptmann.  Lessons for the Future from a Decade of Informedia Video Analysis Research , 2005, CIVR.