Video browsing interfaces and applications: a review

We present a comprehensive review of the state of the art in video browsing and retrieval systems, with special emphasis on interfaces and applications. There has been a significant increase in activity (e.g., storage, retrieval, and sharing) employing video data in the past decade, both for personal and professional use. The ever-growing amount of video content available for human consumption and the inherent characteristics of video data-which, if presented in its raw format, is rather unwieldy and costly-have become driving forces for the development of more effective solutions to present video contents and allow rich user interaction. As a result, there are many contemporary research efforts toward developing better video browsing solutions, which we summarize. We review more than 40 different video browsing and retrieval interfaces and classify them into three groups: applications that use video-player-like interaction, video retrieval applications, and browsing solutions based on video surrogates. For each category, we present a summary of existing work, highlight the technical aspects of each solution, and compare them against each other.

[1]  Pierre Dragicevic,et al.  Video browsing by direct manipulation , 2008, CHI.

[2]  Tat-Seng Chua,et al.  VisionGo: bridging users and multimedia video retrieval , 2008, CIVR '08.

[3]  Frank Hopfgartner,et al.  Semantic User Modelling for Personal News Video Retrieval , 2010, MMM.

[4]  S. Shipman,et al.  Augmenting fast-forward and rewind for personal digital video recorders , 2005, 2005 Digest of Technical Papers. International Conference on Consumer Electronics, 2005. ICCE..

[5]  Werner Bailer,et al.  Video Content Browsing Based on Iterative Feature Clustering for Rushes Exploitation , 2006, TRECVID.

[6]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[7]  Klaus Schöffmann,et al.  Enhancing Seeker-Bars of Video Players with Dominant Color Rivers , 2010, MMM.

[8]  Wolfgang Hürst,et al.  Interactive video browsing on mobile devices , 2007, ACM Multimedia.

[9]  Pierangelo Migliorati,et al.  Interactive visualization of video content and associated description for semantic annotation , 2009, Signal Image Video Process..

[10]  Robert Villa,et al.  FacetBrowser: a user interface for complex search tasks , 2008, ACM Multimedia.

[11]  Sang Uk Lee,et al.  Scalable hierarchical video summary and search , 2001, IS&T/SPIE Electronic Imaging.

[12]  Gary Marchionini,et al.  Video browsing interfaces for the open video project , 2002, CHI Extended Abstracts.

[13]  Klaus Schöffmann,et al.  Video Browsing Using Interactive Navigation Summaries , 2009, 2009 Seventh International Workshop on Content-Based Multimedia Indexing.

[14]  Boon-Lock Yeo,et al.  Video visualization for compact presentation and fast browsing of pictorial content , 1997, IEEE Trans. Circuits Syst. Video Technol..

[15]  Frank Hopfgartner,et al.  Capturing Multiple Interests in News Video Retrieval by Incorporating the Ostensive Model , 2008, PersDB.

[16]  Boon-Lock Yeo,et al.  Classification, simplification, and dynamic visualization of scene transition graphs for video browsing , 1997, Electronic Imaging.

[17]  L.A. Ronningen,et al.  Video browsing techniques for web interfaces , 2006, CCNC 2006. 2006 3rd IEEE Consumer Communications and Networking Conference, 2006..

[18]  Amanda Spink,et al.  From Highly Relevant to Not Relevant: Examining Different Regions of Relevance , 1998, Inf. Process. Manag..

[19]  Regunathan Radhakrishnan,et al.  Video Summarization Using Mpeg-7 Motion Activity and Audio Descriptors , 2003 .

[20]  Steve Benford,et al.  EmoPlayer: A media player for video clips with affective annotations , 2008, Interact. Comput..

[21]  Marti A. Hearst.  Search User Interfaces , 2009 .

[22]  Steven M. Drucker,et al.  SmartSkip: consumer level browsing and skipping of digital video content , 2002, CHI.

[23]  Lawrence Wai-Choong Wong,et al.  ANSES: Summarisation of News Video , 2003, CIVR.

[24]  Marco Ceccarelli,et al.  The color browser: a content driven linear video browsing tool , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[25]  Gary Marchionini,et al.  Effects of audio and visual surrogates for making sense of digital video , 2007, CHI.

[26]  Bing-Yu Chen,et al.  SmartPlayer: user-centric video fast-forwarding , 2009, CHI.

[27]  B. Yegnanarayana,et al.  NVIBRS - news video indexing, browsing and retrieval system , 2005, Proceedings of 2005 International Conference on Intelligent Sensing and Information Processing, 2005..

[28]  Anoop Gupta,et al.  Browsing digital video , 2000, CHI.

[29]  Darren Leigh,et al.  DiamondTouch: a multi-user touch technology , 2001, UIST '01.

[30]  Marcel Worring,et al.  A Learned Lexicon-Driven Paradigm for Interactive Video Retrieval , 2007, IEEE Transactions on Multimedia.

[31]  Wolfgang Hürst,et al.  A new interface for video browsing on PDAs , 2007, Mobile HCI.

[32]  Mark Sanderson,et al.  Advantages of query biased summaries in information retrieval , 1998, SIGIR '98.

[33]  Gerard Salton,et al.  Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer , 1989 .

[34]  Hans-Peter Kriegel,et al.  State-of-the-Art in Content-Based Image and Video Retrieval , 2001, Computational Imaging and Vision.

[35]  Dragutin Petkovic,et al.  "What is in that Video Anyway?" In Search of Better Browsing , 1999, ICMCS, Vol. 1.

[36]  Mark T. Maybury,et al.  Towards content-based browsing of broadcast news video , 1997 .

[37]  João Magalhães,et al.  Video Retrieval Using Search and Browsing , 2004, TRECVID.

[38]  Anita Komlodi,et al.  Visual video browsing interfaces using key frames , 1998, CHI Conference Summary.

[39]  Boon-Lock Yeo,et al.  Extracting story units from long programs for video browsing and navigation , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[40]  Andreas Nürnberger,et al.  VideoSOM: A SOM-Based Interface for Video Browsing , 2006, CIVR.

[41]  Riccardo Leonardi,et al.  The Future-Viewer visual environment for semantic characterization of video sequences , 2005, IEEE International Conference on Image Processing 2005.

[42]  Ramin Zabih,et al.  Video browsing using edges and motion , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[43]  Yasuyuki Nakajima.  A Video Browsing Using Fast Scene Cut Detection for an Efficient Networked Video Database Access (Special Issue on Networked Reality) , 1994 .

[44]  Marcel Worring,et al.  Query on demand video browsing , 2007, ACM Multimedia.

[45]  John R. Kender,et al.  VAST MM: multimedia browser for presentation video , 2007, CIVR '07.

[46]  A. Divakaran,et al.  A Video-Browsing-Enhanced Personal Video Recorder , 2007, 14th International Conference of Image Analysis and Processing - Workshops (ICIAPW 2007).

[47]  Gary Marchionini,et al.  How fast is too fast? evaluating fast forward surrogates for digital video , 2003, 2003 Joint Conference on Digital Libraries, 2003. Proceedings..

[48]  Wei-Ying Ma,et al.  Multimedia information retrieval: what is it, and why isn't anyone using it? , 2005, MIR '05.

[49]  M. Smith,et al.  Video Skimming for Quick Browsing based on Audio and Image Characterization , 1995 .

[50]  M. Flynn,et al.  Context-based video browsing interface for PVR-enabled high-definition television systems , 2006, 2006 Digest of Technical Papers International Conference on Consumer Electronics.

[51]  Jonathan J. Hull,et al.  A paper-based interface for video browsing and retrieval , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[52]  Rong Yan,et al.  Extreme video retrieval: joint maximization of human and computer performance , 2006, MM '06.

[53]  Ryen W. White,et al.  A task-oriented study on the influencing effects of query-biased summarisation in web searching , 2003, Inf. Process. Manag..

[54]  Gary Marchionini,et al.  Alternative Surrogates for Video Objects in a Digital Library: Users' Perspectives on Their Relative Usability , 2002, ECDL.

[55]  Gary Marchionini,et al.  Multimodal surrogates for video browsing , 1999, DL '99.

[56]  Desmond Elliott,et al.  Aspect-based video browsing — A user study , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[57]  Boon-Lock Yeo,et al.  Video browsing using clustering and scene transitions on compressed sequences , 1995, Electronic Imaging.

[58]  B. S. Manjunath,et al.  Content-based search of video using color, texture, and motion , 1997, Proceedings of International Conference on Image Processing.

[59]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[60]  Denis Pellerin,et al.  Interactive Visualization Tool with Graphic Table of Video Contents , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[61]  John P. Oakley,et al.  Storage and Retrieval for Image and Video Databases , 1993 .

[62]  Willemijn Heeren,et al.  Videotrees: Improving video surrogate presentation using hierarchy , 2008, 2008 International Workshop on Content-Based Multimedia Indexing.

[63]  John Adcock,et al.  Experiments in interactive video search by addition and subtraction , 2008, CIVR '08.

[64]  Abby Goodrum.  Multidimensional scaling of video surrogates , 2001, J. Assoc. Inf. Sci. Technol..

[65]  Frank M. Shipman,et al.  Trailblazing: Video Playback Control by Direct Object Manipulation , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[66]  Wolfgang Hürst,et al.  Interactive, dynamic video browsing with the zoomslider interface , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[67]  Shih-Fu Chang,et al.  Clustering methods for video browsing and annotation , 1996, Electronic Imaging.

[68]  Gary Marchionini,et al.  Comprehension and Object Recognition Capabilities for Presentations of Simultaneous Video Key Frame Surrogates , 1997, ECDL.

[69]  Horst Eidenberger.  A Video Browsing Application based on visual MPEG-7 Descriptors and Self-Organising Maps , 2004 .

[70]  Martin Szummer,et al.  Indoor-outdoor image classification , 1998, Proceedings 1998 IEEE International Workshop on Content-Based Access of Image and Video Database.

[71]  Rune Hjelsvold,et al.  Searching and browsing a shared video database , 1995, Proceedings. International Workshop on Multi-Media Database Management Systems.

[72]  Stephen W. Smoliar,et al.  An integrated system for content-based video retrieval and browsing , 1997, Pattern Recognit..

[73]  Yoshinobu Tonomura,et al.  Video browsing using brightness data , 1991, Other Conferences.

[74]  Werner Bailer,et al.  A Tool Supporting Annotation and Analysis of Videos , 2007 .

[75]  Sanjeev Khudanpur,et al.  Imperial College and Johns Hopkins University at TRECVID , 2006, TRECVID.

[76]  Yasuyuki Matsushita,et al.  Interactive video exploration using pose slices , 2006, SIGGRAPH '06.

[77]  Martin Halvey,et al.  ViGOR: a grouping oriented interface for search and retrieval in video libraries , 2009, JCDL '09.

[78]  Dragutin Petkovic,et al.  Key to effective video retrieval: effective cataloging and browsing , 1998, MULTIMEDIA '98.

[79]  Clifton Forlines,et al.  Rapid serial visual presentation techniques for consumer digital video devices , 2003, UIST '03.

[80]  Gary Marchionini,et al.  Key frame preview techniques for video browsing , 1998, DL '98.

[81]  Remi Depommier,et al.  Content-based browsing of video sequences , 1994, MULTIMEDIA '94.

[82]  Zhang Hui,et al.  A content-based broadcasted sports video retrieval system using multiple modalities: SportBR , 2005, The Fifth International Conference on Computer and Information Technology (CIT'05).

[83]  Michael G. Christel.  Supporting video library exploratory search: when storyboards are not enough , 2008, CIVR '08.

[84]  Neema Moraveji.  Improving video browsing with an eye-tracking evaluation of feature-based color bars , 2004, Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004..

[85]  Xinbo Gao,et al.  NewsEye: a news video browsing and retrieval system , 2001, Proceedings of 2001 International Symposium on Intelligent Multimedia, Video and Speech Processing. ISIMP 2001 (IEEE Cat. No.01EX489).

[86]  Timo Ojala,et al.  Cluster-temporal browsing of large news video databases , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[87]  Xian-Sheng Hua,et al.  Smart video player , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[88]  Stephen W. Smoliar,et al.  Video parsing, retrieval and browsing: an integrated and content-based solution , 1997, MULTIMEDIA '95.

[89]  Klaus Schöffmann,et al.  Video browsing using motion visualization , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[90]  Stephen W. Smoliar,et al.  Content-based video browsing tools , 1995, Electronic Imaging.

[91]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[92]  Timo Ojala,et al.  TRECVID 2005 Experiments at Media Team Oulu , 2005, TRECVID.

[93]  Nicholas J. Belkin,et al.  Braque: Design of an Interface to Support User Interaction in Information Retrieval , 1993, Inf. Process. Manag..

[94]  Gary Marchionini,et al.  Dynamic key frame presentation techniques for augmenting video browsing , 1998, AVI '98.

[95]  John R. Smith,et al.  IBM Research TRECVID-2009 Video Retrieval System , 2009, TRECVID.

[96]  Anil K. Jain,et al.  Image classification for content-based indexing , 2001, IEEE Trans. Image Process..

[97]  Tobias Lauer,et al.  New methods for visual information seeking through video browsing , 2004, Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004..

[98]  Svetha Venkatesh,et al.  Temporal semantic compression for video browsing , 2008, IUI '08.

[99]  Riad Hammoud,et al.  Interactive Video: Algorithms and Technologies (Signals and Communication Technology) , 2006 .

[100]  Wolfgang Hürst.  Interactive audio-visual video browsing , 2006, MM '06.

[101]  Riccardo Leonardi,et al.  An Intuitive Graphic Environment for Navigation and Classification of Multimedia Documents , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[102]  Ajay Divakaran,et al.  Adaptive fast playback-based video skimming using a compressed-domain visual complexity measure , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[103]  John S. Boreczky,et al.  Comparison of video shot boundary detection techniques , 1996, J. Electronic Imaging.

[104]  Alan F. Smeaton,et al.  TRECVid 2006 Experiments at Dublin City University , 2012, TRECVID.

[105]  Alan F. Smeaton,et al.  User evaluation of Físchlár-News , 2006, ACM Trans. Inf. Syst..

[106]  Yanxiang He,et al.  NewsBR: a content-based news video browsing and retrieval system , 2004, The Fourth International Conference onComputer and Information Technology, 2004. CIT '04..

[107]  Gary Marchionini,et al.  Text or Pictures? An Eyetracking Study of How People View Digital Video Surrogates , 2003, CIVR.

[108]  Dennis Koelma,et al.  The MediaMill TRECVID 2008 Semantic Video Search Engine , 2008, TRECVID.