Search in audiovisual broadcast archives

Documentary makers, journalists, news editors, and other media professionals routinely require previously recorded audiovisual material for new productions. For example, a news editor might wish to reuse footage shot by overseas services for the evening news, or a documentary maker might require shots of Christmas trees recorded over the decades. Important sources for reusable broadcasts are audiovisual broadcast archives, which preserve and manage audiovisual material. With digitization, media professional can be given online access to video. This increases ease of access, but increases the need for search capabilities tailored for the media professional. Search in audiovisual broadcast archives, then, is the subject of this thesis. We begin by investigating the search behavior of media professionals in current daily practice. To this end we perform a large-scale log analysis of their search actions at a national audiovisual broadcast archive. Our analysis characterizes not only the searches of media professionals, but also their purchasing behavior. In order to model the observed behavior we follow our log analysis with a simulation experiment. Here we investigate simulation methods for recreating the searches and purchases recorded in the archive to create evaluation testbeds. In the second half of the thesis we turn to investigate the use of state-of-art methods for retrieval with automatically generated content metadata from video, Specifically we focus on their application for improving audiovisual fragment search in the audiovisual broadcast archive. We use logged searches and purchases to define new test collections for retrieval evaluation. These are used as the basis for experiments aimed at solving specific problems that are faced when searching with automatically generated descriptions of video content. Finally, we combine state-of-the-art methods with the current daily practice of the archive, and investigate their potential combined impact on search in audiovisual broadcast archives. The contributions of this thesis include the characterization of searching and purchasing behaviour of media professionals at a large audiovisual broadcast archive, and a framework for simulating their logged queries and purchases. Contributions in the second half of the thesis include an in-depth user study of how text queries should be mapped to visual concepts, a retrieval model that accounts for the temporal mismatch between the speech and visual tracks in audiovisual material, and a set of experiments demonstrating the effectiveness of automatically generated content metadata for improving retrieval in the audiovisual broadcast archive. The thesis can be accessed at http://dare.uva.nl/record/358972.

[1]  Bernard J. Jansen,et al.  The Methodology of Search Log Analysis , 2009 .

[2]  Morten Hertzum,et al.  Requests for information from a film archive: a case study of multimedia retrieval , 2003, J. Documentation.

[3]  Milind R. Naphade,et al.  Semantic Multimedia Retrieval using Lexical Query Expansion and Model-Based Reranking , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[4]  Marcel Worring,et al.  Multimodal Video Indexing : A Review of the State-ofthe-art , 2001 .

[5]  Djoerd Hiemstra,et al.  Reusing annotation labor for concept selection , 2009, CIVR '09.

[6]  James Allan,et al.  Approaches to passage retrieval in full text information systems , 1993, SIGIR.

[7]  Peter Wilkins,et al.  An investigation into weighted data fusion for content-based multimedia information retrieval , 2009 .

[8]  Amanda Spink,et al.  The Effect of Specialized Multimedia Collections on Web Searching , 2004, J. Web Eng..

[9]  Marcel Worring,et al.  VideOlympics: Real-Time Evaluation of Multimedia Retrieval Systems , 2008, IEEE MultiMedia.

[10]  Mark Sanderson,et al.  Test Collection Based Evaluation of Information Retrieval Systems , 2010, Found. Trends Inf. Retr..

[11]  Christian Petersohn.  Fraunhofer HHI at TRECVID 2004: Shot Boundary Detection System , 2004, TRECVID.

[12]  Rong Yan,et al.  Probabilistic latent query analysis for combining multiple retrieval sources , 2006, SIGIR.

[13]  Dong Wang,et al.  Video search in concept subspace: a text-like paradigm , 2007, CIVR '07.

[14]  Alan F. Smeaton,et al.  Properties of optimally weighted data fusion in CBMIR , 2010, SIGIR.

[15]  Marcel Worring,et al.  Balancing thread based navigation for targeted video search , 2008, CIVR '08.

[16]  Maarten de Rijke,et al.  Search behavior of media professionals at an audiovisual archive: A transaction log analysis , 2010, J. Assoc. Inf. Sci. Technol..

[17]  Martha Larson,et al.  Overview of VideoCLEF 2008: Automatic Generation of Topic-based Feeds for Dual Language Audio-Visual Content , 2008, CLEF.

[18]  Yanjun Qi,et al.  Video Classification and Retrieval with the Informedia Digital Video Library System , 2002, TREC.

[19]  Wei-Hao Lin,et al.  Which Thousand Words are Worth a Picture? Experiments on Video Retrieval using a Thousand Concepts , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[20]  Ellen M. Voorhees,et al.  The Philosophy of Information Retrieval Evaluation , 2001, CLEF.

[21]  Frank Hopfgartner,et al.  Evaluating the implicit feedback models for adaptive video retrieval , 2007, MIR '07.

[22]  W. V. D. van den Heuvel,et al.  Expert search for radio and television: a case study amongst Dutch broadcast professionals , 2010 .

[23]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[24]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[25]  Michael G. Christel.  Establishing the utility of non-text search for news video retrieval with real world users , 2007, ACM Multimedia.

[26]  Michael D. Gordon.  Evaluating the effectiveness of information retrieval systems using simulated queries , 1990, J. Am. Soc. Inf. Sci..

[27]  Marcel Worring,et al.  Adding Semantics to Detectors for Video Retrieval , 2007, IEEE Transactions on Multimedia.

[28]  Jean Tague-Sutcliffe,et al.  Problems in the simulation of bibliographic retrieval systems , 1980, SIGIR '80.

[29]  Bernard Cole.  Search engines tackle the desktop , 2005, Computer.

[30]  Amanda Spink,et al.  A study and comparison of multimedia Web searching: 1997-2006 , 2009, J. Assoc. Inf. Sci. Technol..

[31]  Rong Yan,et al.  Probabilistic models for combining diverse knowledge sources in multimedia retrieval , 2006 .

[32]  Alan F. Smeaton,et al.  TRECVid 2006 Experiments at Dublin City University , 2012, TRECVID.

[33]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[34]  Jun Yang,et al.  Finding Person X: Correlating Names with Visual Appearances , 2004, CIVR.

[35]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Alan Hanjalic,et al.  Shot-boundary detection: unraveled and resolved? , 2002, IEEE Trans. Circuits Syst. Video Technol..

[37]  Dennis Koelma,et al.  The MediaMill TRECVID 2008 Semantic Video Search Engine , 2008, TRECVID.

[38]  Véronique Malaisé,et al.  A Web Based General Thesaurus Browser to Support Indexing of Television and Radio Programs , 2006, LREC.

[39]  Philip Resnik,et al.  Using Information Content to Evaluate Semantic Similarity in a Taxonomy , 1995, IJCAI.

[40]  Rong Yan,et al.  Learning query-class dependent weights in automatic video retrieval , 2004, MULTIMEDIA '04.

[41]  Nevenka Dimitrova.  Multimedia Content Analysis: The Next Wave , 2003, CIVR.

[42]  Bernard J. Jansen,et al.  A review of Web searching studies and a framework for future research , 2001, J. Assoc. Inf. Sci. Technol..

[43]  Marcel Worring,et al.  Query on demand video browsing , 2007, ACM Multimedia.

[44]  Amit Singhal,et al.  Document expansion for speech retrieval , 1999, SIGIR '99.

[45]  Meng Wang,et al.  MSRA atT TRECVID 2008: High-Level Feature Extraction and Automatic Search , 2008, TRECVID.

[46]  Karen Spärck Jones,et al.  Automatic content-based retrieval of broadcast news , 1995, MULTIMEDIA '95.

[47]  Shih-Fu Chang,et al.  A reranking approach for context-based concept fusion in video indexing and retrieval , 2007, CIVR '07.

[48]  Tao Tao,et al.  Language Model Information Retrieval with Document Expansion , 2006, NAACL.

[49]  Filip Radlinski,et al.  How does clickthrough data reflect retrieval quality? , 2008, CIKM '08.

[50]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[51]  Corinne Jörgensen,et al.  Image querying by image professionals , 2005, J. Assoc. Inf. Sci. Technol..

[52]  Franciska de Jong,et al.  Multimedia Search Without Visual Analysis: The Value of Linguistic and Contextual Information , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[53]  Katja Hofmann,et al.  Comparing click-through data to purchase decisions for retrieval evaluation , 2010, SIGIR '10.

[54]  Yihong Gong,et al.  Lessons Learned from Building a Terabyte Digital Video Library , 1999, Computer.

[55]  Javed A. Aslam,et al.  Models for metasearch , 2001, SIGIR '01.

[56]  Maarten de Rijke,et al.  Exploiting redundancy in cross-channel video retrieval , 2007, MIR '07.

[57]  Leif Azzopardi.  Query side evaluation: an empirical analysis of effectiveness and effort , 2009, SIGIR.

[58]  CHENGXIANG ZHAI,et al.  A study of smoothing methods for language models applied to information retrieval , 2004, TOIS.

[59]  Alexander G. Hauptmann.  Lessons for the Future from a Decade of Informedia Video Analysis Research , 2005, CIVR.

[60]  Marcel Worring,et al.  Concept-Based Video Retrieval , 2009, Found. Trends Inf. Retr..

[61]  Thierry Urruty,et al.  Simulated evaluation of faceted browsing based on feature selection , 2010, Multimedia Tools and Applications.

[62]  John R. Smith,et al.  Large-scale concept ontology for multimedia , 2006, IEEE MultiMedia.

[63]  Ellen M. Voorhees,et al.  Evaluating evaluation measure stability , 2000, SIGIR '00.

[64]  George Hripcsak,et al.  Technical Brief: Agreement, the F-Measure, and Reliability in Information Retrieval , 2005, J. Am. Medical Informatics Assoc..

[65]  Stephen W. Smoliar,et al.  Content based video indexing and retrieval , 1994, IEEE MultiMedia.

[66]  Qigang Gao,et al.  Using controlled query generation to evaluate blind relevance feedback algorithms , 2006, Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '06).

[67]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[68]  Ross Wilkinson,et al.  Effective retrieval of structured documents , 1994, SIGIR '94.

[69]  Jean-Luc Gauvain,et al.  Modeling northern and southern varieties of dutch for STT , 2009, INTERSPEECH.

[70]  Alan F. Smeaton,et al.  TRECVID 2004 Experiments in Dublin City University , 2004, TRECVID.

[71]  Sheng Tang,et al.  TRECVID 2006 by NUS-I2R , 2006, TRECVID.

[72]  Katja Hofmann,et al.  Assessing concept selection for video retrieval , 2008, MIR '08.

[73]  Amanda Spink,et al.  Multimedia Web searching trends: 1997-2001 , 2003, Inf. Process. Manag..

[74]  Thorsten Joachims,et al.  Accurately interpreting clickthrough data as implicit feedback , 2005, SIGIR '05.

[75]  J. D. Bernal,et al.  The Royal Society Scientific Information Conference , 1948, Nature.

[76]  Desmond Elliott,et al.  Supporting aspect-based video browsing: analysis of a user study , 2009, CIVR '09.

[77]  François Brémond,et al.  ETISEO, performance evaluation for video surveillance systems , 2007, 2007 IEEE Conference on Advanced Video and Signal Based Surveillance.

[78]  Kalervo Järvelin,et al.  Test Collection-Based IR Evaluation Needs Extension toward Sessions - A Case of Extremely Short Queries , 2009, AIRS.

[79]  David Hawking,et al.  Overview of the TREC 2003 Web Track , 2003, TREC.

[80]  Martin Halvey,et al.  The role of expertise in aiding video search , 2009, CIVR '09.

[81]  Rong Yan,et al.  A review of text and image retrieval approaches for broadcast news video , 2007, Information Retrieval.

[82]  Dong Xu,et al.  Columbia University TRECVID-2006 Video Search and High-Level Feature Extraction , 2006, TRECVID.

[83]  Cor J. Veenman,et al.  Episode-Constrained Cross-Validation in Video Concept Retrieval , 2009, IEEE Transactions on Multimedia.

[84]  Laura Hollink,et al.  Search behavior of media professionals at an audiovisual archive: A transaction log analysis , 2010 .

[85]  M VoorheesEllen.  The TREC question answering track , 2001 .

[86]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[87]  Gilad Mishne,et al.  A Study of Blog Search , 2006, ECIR.

[88]  Peter G. B. Enser,et al.  Analysis of user need in image archives , 1997, J. Inf. Sci..

[89]  Ellen M. Voorhees,et al.  Variations in relevance judgments and the measurement of retrieval effectiveness , 1998, SIGIR '98.

[90]  Maarten de Rijke,et al.  Today's and tomorrow's retrieval practice in the audiovisual archive , 2010, CIVR '10.

[91]  Jing Zhang,et al.  Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[92]  R. Edmondson.  Audiovisual archiving : philosophy and principles , 2004 .

[93]  M. de Rijke,et al.  Simulating searches from transaction logs , 2010 .

[94]  Tobun Dorbin Ng,et al.  Informedia at TRECVID 2003 : Analyzing and Searching Broadcast News Video , 2003, TRECVID.

[95]  Kevin Andreano,et al.  The Missing Link: Content Indexing, User-Created Metadata, and Improving Scholarly Access to Moving Image Archives , 2008 .

[96]  John R. Smith,et al.  A web-based system for collaborative annotation of large image and video collections: an evaluation and user study , 2005, MULTIMEDIA '05.

[97]  Alan F. Smeaton,et al.  User variance and its impact on video retrieval benchmarking , 2009, CIVR '09.

[98]  ChengXiang Zhai,et al.  Evaluation of methods for relative comparison of retrieval systems based on clickthroughs , 2009, CIKM.

[99]  Arnold W. M. Smeulders,et al.  Visual-Concept Search Solved? , 2010, Computer.

[100]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[101]  Stephen E. Robertson,et al.  On the history of evaluation in IR , 2008, J. Inf. Sci..

[102]  Peter G. B. Enser,et al.  VIRAMI - Visual Information Retrieval for Archival Moving Imagery , 2001, ICHIM.

[103]  M. de Rijke,et al.  Building simulated queries for known-item topics: an analysis using six european languages , 2007, SIGIR.

[104]  Peter G. B. Enser,et al.  The evolution of visual information retrieval , 2008, J. Inf. Sci..

[105]  Alan F. Smeaton,et al.  A Comparison of Score, Rank and Probability-Based Fusion Methods for Video Shot Retrieval , 2005, CIVR.

[106]  Stephen W. Smoliar,et al.  An integrated system for content-based video retrieval and browsing , 1997, Pattern Recognit..

[107]  Ellen M. Voorhees,et al.  The TREC Spoken Document Retrieval Track: A Success Story , 2000, TREC.

[108]  Shih-Fu Chang,et al.  Visually Searching the Web for Content , 1997, IEEE Multim..

[109]  Gary Marchionini,et al.  Information Seeking in Electronic Environments , 1995 .

[110]  Alan F. Smeaton.  Techniques used and open challenges to the analysis, indexing and retrieval of digital video , 2007, Inf. Syst..

[111]  Rong Yan,et al.  Can High-Level Concepts Fill the Semantic Gap in Video Retrieval? A Case Study With Broadcast News , 2007, IEEE Transactions on Multimedia.

[112]  Ronald E. Rice,et al.  The use of computer-monitored data in information science and communication research , 1983, J. Am. Soc. Inf. Sci..

[113]  Marcel Worring,et al.  The challenge problem for automated detection of 101 semantic concepts in multimedia , 2006, MM '06.

[114]  Thijs Westerveld,et al.  Using generative probabilistic models for multimedia retrieval , 2005, SIGF.

[115]  Shih-Fu Chang,et al.  Video search reranking via information bottleneck principle , 2006, MM '06.

[116]  Franciska de Jong,et al.  Annotation of Heterogeneous Multimedia Content Using Automatic Speech Recognition , 2007, SAMT.

[117]  Marcel Worring,et al.  Browsing Video Along Multiple Threads , 2010, IEEE Transactions on Multimedia.

[118]  Koen E. A. van de Sande,et al.  Evaluating Color Descriptors for Object and Scene Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[119]  Maarten de Rijke,et al.  Term Selection and Query Operations for Video Retrieval , 2007, ECIR.

[120]  Christof Monz,et al.  The QMUL system description for IWSLT 2010 , 2010, IWSLT.

[121]  George Lakoff,et al.  Women, Fire, and Dangerous Things , 1987 .

[122]  Jin Zhao,et al.  Video Retrieval Using High Level Features: Exploiting Query Matching and Confidence-Based Weighting , 2006, CIVR.

[123]  Dong Wang,et al.  The importance of query-concept-mapping for automatic video retrieval , 2007, ACM Multimedia.

[124]  Massimo Barbaro,et al.  A Face Is Exposed for AOL Searcher No , 2006 .

[125]  Marcel Worring,et al.  The Semantic Pathfinder: Using an Authoring Metaphor for Generic Multimedia Indexing , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[126]  Paul Solomon,et al.  Looking for Information—A Survey of Research on Information Seeking, Needs, and Behavior , 2003, Information Retrieval.

[127]  Shih-Fu Chang,et al.  Automatic discovery of query-class-dependent models for multimodal search , 2005, MULTIMEDIA '05.

[128]  John R. Smith,et al.  Integrating Features, Models, and Semantics for TREC Video Retrieval , 2001, TREC.

[129]  Rong Yan,et al.  Negative pseudo-relevance feedback in content-based video retrieval , 2003, MULTIMEDIA '03.

[130]  Christopher D. Manning,et al.  Introduction to Information Retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[131]  Zhu Liu,et al.  Multimedia content analysis-using both audio and visual clues , 2000, IEEE Signal Process. Mag..

[132]  Michael A. Shepherd,et al.  A Field Study Characterizing Web-based Information Seeking Tasks , 2022 .

[133]  Hung-Khoon Tan,et al.  Beyond Semantic Search: What You Observe May Not Be What You Think , 2008, TRECVID.

[134]  Djoerd Hiemstra,et al.  Using language models for information retrieval , 2001 .

[135]  M. de Rijke,et al.  Learning Semantic Query Suggestions , 2009, SEMWEB.

[136]  Djoerd Hiemstra,et al.  Probabilistic Approaches to Video Retrieval , 2004, TRECVID.

[137]  Wei-Ying Ma,et al.  Optimizing web search using web click-through data , 2004, CIKM '04.

[138]  Maarten de Rijke,et al.  Shallow Morphological Analysis in Monolingual Information Retrieval for Dutch, German, and Italian , 2001, CLEF.

[139]  Amanda Spink,et al.  Defining a session on Web search engines , 2007, J. Assoc. Inf. Sci. Technol..

[140]  Paul Over,et al.  TRECVID: evaluating the effectiveness of information retrieval tasks on digital video , 2004, MULTIMEDIA '04.

[141]  John R. Smith,et al.  IBM Research TRECVID-2009 Video Retrieval System , 2009, TRECVID.

[142]  Paul Over,et al.  TRECVID 2007--Overview , 2007, TRECVID.

[143]  Xian-Sheng Hua,et al.  Bayesian video search reranking , 2008, ACM Multimedia.

[144]  Shih-Fu Chang,et al.  VideoQ: an automated content based video search system using visual cues , 1997, MULTIMEDIA '97.

[145]  Amanda Spink,et al.  Real life, real users, and real needs: a study and analysis of user queries on the web , 2000, Inf. Process. Manag..

[146]  Nicu Sebe,et al.  Content-based multimedia information retrieval: State of the art and challenges , 2006, TOMCCAP.

[147]  Katja Hofmann,et al.  A Semantic Perspective on Query Log Analysis , 2009, CLEF.

[148]  Katja Hofmann,et al.  Validating Query Simulators: An Experiment Using Commercial Searches and Purchases , 2010, CLEF.

[149]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[150]  Amanda Spink,et al.  Determining the informational, navigational, and transactional intent of Web queries , 2008, Inf. Process. Manag..

[151]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[152]  Tom Peters,et al.  The history and development of transaction log analysis , 1993 .

[153]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[154]  M. Bloor.  The Structures of the Life-World , 1975 .

[155]  H. Bourlard,et al.  Interpretation of Multiparty Meetings the AMI and Amida Projects , 2008, 2008 Hands-Free Speech Communication and Microphone Arrays.

[156]  Jonathan Foote,et al.  An overview of audio information retrieval , 1999, Multimedia Systems.

[157]  Chong-Wah Ngo,et al.  Ontology-enriched semantic space for video search , 2007, ACM Multimedia.

[158]  Rong Yan,et al.  Semantic concept-based query expansion and re-ranking for multimedia retrieval , 2007, ACM Multimedia.

[159]  Daniel E. Rose,et al.  Understanding user goals in web search , 2004, WWW '04.

[160]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[161]  Maarten de Rijke,et al.  The value of stories for speech-based video search , 2007, CIVR '07.

[162]  Jun Yang,et al.  Exploring temporal consistency for video analysis and retrieval , 2006, MIR '06.

[163]  Andrei Broder,et al.  A taxonomy of web search , 2002, SIGF.

[164]  Djoerd Hiemstra,et al.  The Effectiveness of Concept Based Search for Video Retrieval , 2007, LWA.

[165]  Alexander G. Hauptmann,et al.  The Use and Utility of High-Level Semantic Features in Video Retrieval , 2005, CIVR.

[166]  Rong Yan,et al.  Extreme video retrieval: joint maximization of human and computer performance , 2006, MM '06.

[167]  Yi-Hsuan Yang,et al.  Video search reranking via online ordinal reranking , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[168]  Erwin Panofsky,et al.  Studies in Iconology , 1962 .

[169]  Martin Halvey,et al.  Search trails using user feedback to improve video search , 2008, ACM Multimedia.

[170]  Fabio Crestani,et al.  A statistical comparison of tag and query logs , 2009, SIGIR.

[171]  David M. Nichols,et al.  How people find videos , 2008, JCDL '08.

[172]  Gang Wang,et al.  TRECVID 2004 Search and Feature Extraction Task by NUS PRIS , 2004, TRECVID.

[173]  W. Bruce Croft,et al.  Query reformulation using anchor text , 2010, WSDM '10.

[174]  Jean Tague-Sutcliffe,et al.  Simulation of User Judgments in Bibliographic Retrieval Systems , 1981, SIGIR.

[175]  Paul Over,et al.  TRECVID 2005 - An Overview , 2005, TRECVID.

[176]  Marcel Worring,et al.  Content‐based video retrieval: Three example systems from TRECVid , 2008, Int. J. Imaging Syst. Technol..