Simulating the future of concept-based video retrieval under improved detector performance

In this paper we address the following important questions for concept-based video retrieval: (1) What is the impact of detector performance on the performance of concept-based retrieval engines, and (2) will these engines be applicable to real-life search tasks if detector performance improves in the future? We use Monte Carlo simulations to answer these questions. To generate the simulation input, we propose to use a probabilistic model of two Gaussians for the confidence scores that concept detectors emit. Modifying the model’s parameters affects the detector performance and the search performance. We study the relation between these two performances on two video collections. For detectors with similar discriminative power and a concept vocabulary of around 100 concepts, the simulation reveals that in order to achieve a search performance of 0.20 mean average precision (MAP)—which is considered sufficient performance for real-life applications—one needs detectors with at least 0.60 MAP . We also find that, given our simulation model and low detector performance, MAP is not always a good evaluation measure for concept detectors since it is not strongly correlated with the search performance.

[1]  John A. Bather,et al.  Decision Theory: An Introduction to Dynamic Programming and Sequential Decisions , 2000 .

[2]  Alexander G. Hauptmann,et al.  The Use and Utility of High-Level Semantic Features in Video Retrieval , 2005, CIVR.

[3]  Chong-Wah Ngo,et al.  Representations of Keypoint-Based Semantic Concept Detection: A Comprehensive Study , 2010, IEEE Transactions on Multimedia.

[4]  Sheldon M. Ross,et al.  Introduction to Probability Models, Eighth Edition , 1972 .

[5]  John R. Smith,et al.  Large-scale concept ontology for multimedia , 2006, IEEE MultiMedia.

[6]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[7]  W. Press,et al.  Numerical Recipes in C++: The Art of Scientific Computing (2nd edn)1 Numerical Recipes Example Book (C++) (2nd edn)2 Numerical Recipes Multi-Language Code CD ROM with LINUX or UNIX Single-Screen License Revised Version3 , 2003 .

[8]  J. Taylor.  An Introduction to Error Analysis , 1982 .

[9]  Marcel Worring,et al.  Concept-Based Video Retrieval , 2009, Found. Trends Inf. Retr..

[10]  Sheldon M. Ross,et al.  Introduction to probability models , 1975 .

[11]  Alexander G. Hauptmann,et al.  SPEECH RECOGNITION AND INFORMATION RETRIEVAL: EXPERIMENTS IN RETRIEVING SPOKEN DOCUMENTS , 1997 .

[12]  Marcel Worring,et al.  The challenge problem for automated detection of 101 semantic concepts in multimedia , 2006, MM '06.

[13]  Rong Yan,et al.  Probabilistic models for combining diverse knowledge sources in multimedia retrieval , 2006 .

[14]  W. B. Croft,et al.  An Evaluation of Information Retrieval Accuracy with Simulated OCR Output , 1993 .

[15]  Anawach Sangswang,et al.  Justification of a stochastic model for a DC-DC boost converter , 2003, IECON'03. 29th Annual Conference of the IEEE Industrial Electronics Society (IEEE Cat. No.03CH37468).

[16]  Mohan S. Kankanhalli,et al.  Proceedings of the 2008 international conference on Content-based image and video retrieval , 2008 .

[17]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[18]  Dennis Koelma,et al.  The MediaMill TRECVID 2008 Semantic Video Search Engine , 2008, TRECVID.

[19]  Marcel Worring,et al.  Are Concept Detector Lexicons Effective for Video Search? , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[20]  Rong Yan,et al.  The combination limit in multimedia retrieval , 2003, MULTIMEDIA '03.

[21]  Djoerd Hiemstra,et al.  Reusing annotation labor for concept selection , 2009, CIVR '09.

[22]  Christoph Arndt,et al.  Information Measures: Information and its Description in Science and Engineering , 2001 .

[23]  Jun Yang,et al.  (Un)Reliability of video concept detection , 2008, CIVR '08.

[24]  Djoerd Hiemstra,et al.  A probabilistic ranking framework using unobservable binary events for video search , 2008, CIVR '08.

[25]  Rong Yan,et al.  Can High-Level Concepts Fill the Semantic Gap in Video Retrieval? A Case Study With Broadcast News , 2007, IEEE Transactions on Multimedia.

[26]  David Hawking,et al.  Overview of the TREC-9 Web Track , 2000, TREC.

[27]  Stéphane Ayache,et al.  Evaluation of active learning strategies for video indexing , 2007, Signal Process. Image Commun..

[28]  Alan F. Smeaton,et al.  Measuring the Influence of Concept Detection on Video Retrieval , 2009, CAIP.

[29]  Djoerd Hiemstra,et al.  Concept detectors: how good is good enough? , 2009, ACM Multimedia.

[30]  Paul Over,et al.  High-level feature detection from video in TRECVid: a 5-year retrospective of achievements , 2009 .

[31]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[32]  Stephen E. Robertson,et al.  Probabilistic models of indexing and searching , 1980, SIGIR '80.

[33]  N. Metropolis,et al.  The Monte Carlo method. , 1949 .

[34]  Robert Tibshirani,et al.  Classification by Pairwise Coupling , 1997, NIPS.

[35]  Hsuan-Tien Lin,et al.  A note on Platt’s probabilistic outputs for support vector machines , 2007, Machine Learning.

[36]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[37]  Djoerd Hiemstra,et al.  A Simulator for Concept Detector Output , 2009 .

[38]  Bo Zhang,et al.  Using High-Level Semantic Features in Video Retrieval , 2006, CIVR.

[39]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[40]  Rong Yan,et al.  A review of text and image retrieval approaches for broadcast news video , 2007, Information Retrieval.

[41]  D. J. White,et al.  Decision Theory , 2018, Behavioral Finance for Private Banking.

[42]  Alan F. Smeaton,et al.  A Comparison of Score, Rank and Probability-Based Fusion Methods for Video Shot Retrieval , 2005, CIVR.

[43]  C. J. van Rijsbergen,et al.  Proceedings of the 3rd annual ACM conference on Research and development in information retrieval , 1980 .