With the rapid increase in online video services, multimedia retrieval systems are becoming increasingly important search tools to users in many different fields. In this paper we present a novel retrieval interface, "FacetBrowser", which supports the creation of multiple search "facets", to aid users carrying out complex search tasks involving multiple concepts. Each facet represents a different aspect of the search task: an assumption of this work is that search facets are best represented by sub-searches, providing users with flexibility in defining facets on the fly, rather than using pre-defined categories or metadata information as used in many other exploratory search interfaces [3, 8, 17]. Such facets can be organised into "stories" by users, facilitating users in building up sequences of related searches and material which together can be used to satisfy a work task. The interface allows more than one search to be executed and viewed simultaneously, and importantly, allows material to be reorganized between the facets, acknowledging the inter-relatedness which can often occur between search facets. The design of the FacetBrowser interface is presented, along with an experiment comparing it to a tabbed interface similar to that on modern web browsers. The results suggest that the FacetBrowser has the potential to aid users in exploring and structuring their searching effort when carrying out broad search tasks.

With the rapid growth of multimedia data, it becomes increasingly important to develop semantic concept modeling approaches that are consistently effective, highly efficient, and easily scalable. To this end, we first propose the robust subspace bagging (RB-SBag) algorithm by augmenting random subspace bagging with forward model selection. Compared with traditional modeling approaches, RB-SBag offers a considerably faster learning process while minimizing the risk of overfitting. Its ensemble structure also enables a convenient transformation into a simple parallel framework called MapReduce. To further improve scalability, we also develop a task scheduling algorithm to optimize task placement for heterogenous tasks. On a collection consisting of more than 250,000 images and several standard TRECVID benchmark datasets, RB-SBag achieved more than a 10-fold speedup with comparable or even better classification performance than baseline SVMs. We also deployed the MapReduce implementation on a 16-node Hadoop cluster, where the proposed task scheduler demonstrates a significantly better scalability than the baseline scheduler in the presence of task heterogeneity.

We propose a new interactive image and video retrieval system called multi-query interactive retrieval, which is designed to jointly optimize the retrieval performance on multiple query topics. The proposed system employs a learning-based hybrid retrieval approach, which can automatically switch between tagging and browsing interface based on user labeling efficiency. To formalize the retrieval process, we use two formal annotation models to track and estimate the retrieval time for each method. Based on the parameters of these models, the system integrates the tagging-based and browsing-based methods in order to minimize overall retrieval time across the full set of query topics. This hybrid multi-topic retrieval approach is demonstrated to be highly effective on two large-scale video collections.

We present methods for semantic annotation of multimedia data. The goal is to detect semantic attributes (also referred to as concepts) in clips of video via analysis of a single keyframe or set of frames. The proposed methods integrate high performance discriminative single concept detectors in a random field model for collective multiple concept detection. Furthermore, we describe a generic framework for semantic media classification capable of capturing arbitrary complex dependencies between the semantic concepts. Finally, we present initial experimental results comparing the proposed approach to existing methods.

We present methods for semantic annotation of multimedia data. The goal is to detect semantic attributes (also referred to as concepts) in clips of video via analysis of a single keyframe or set of frames. The proposed methods integrate high performance discriminative single concept detectors in a random field model for collective multiple concept detection. Furthermore, we describe a generic framework for semantic media classification capable of capturing arbitrary complex dependencies between the semantic concepts. Finally, we present initial experimental results comparing the proposed approach to existing methods.

We present a model for exploiting community-based usage information for video retrieval, where implicit usage information from past users is exploited in order to provide enhanced assistance in video retrieval tasks, and alleviate the effects of the semantic gap problem. We propose a graph-based model for all types of implicit and explicit feedback, in which the relevant usage information is represented. Our model is designed to capture the complex interactions of a user with an interactive video retrieval system, including the representation of sequences of user-system interaction during a search session. Building upon this model, four recommendation strategies are defined and evaluated. An evaluation strategy is proposed based on simulated user actions, which enables the evaluation of our recommendation strategies over a usage information pool obtained from 24 users performing four different TRECVid tasks. Furthermore, the proposed simulation approach is used to simulate usage information pools with different characteristics, with which the recommendation approaches are further evaluated on a larger set of tasks, and their performance is studied with respect to the scalability and quality of the available implicit information.

Use of test collections and evaluation measures to assess the effectiveness of information retrieval systems has its origins in work dating back to the early 1950s. Across the nearly 60 years since that work started, use of test collections is a de facto standard of evaluation. This monograph surveys the research conducted and explains the methods and measures devised for evaluation of retrieval systems, including a detailed look at the use of statistical significance testing in retrieval experimentation. This monograph reviews more recent examinations of the validity of the test collection approach and evaluation measures as well as outlining trends in current research exploiting query logs and live labs. At its core, the modern-day test collection is little different from the structures that the pioneering researchers in the 1950s and 1960s conceived of. This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research.

This paper studies the combination of textual and visual information in a database of medical records in order to improve the performance of the multi-modal information retrieval system. The proposed model consists of two subsystems: a content-based information retrieval subsystem that performs the image retrieval and a textual information retrieval subsystem that performs the textual retrieval. The images and text are independently retrieved and then the partial resulting lists are mixed. A study of different weighting schemes has been accomplished and analyzed. The results obtained show that the proper integration of textual information improves conventional multi-modal systems.

This paper investigates new approaches to improve the efficiency of manual image annotation and help users to produce better annotation results in a given amount of time. Although important in practice, this issue has rarely been studied in a quantitative way before. To achieve this, we first propose two time models to analyze the annotation process for two popular manual annotation approaches, i.e., tagging and browsing. The complementary properties of these approaches have inspired us to merge them to develop a hybrid annotation algorithms called frequency-based annotation. Our experiments on large-scale multimedia collections have shown that the proposed algorithm can achieve an up to 40% annotation time reduction compared with the baseline methods. In other words, it can produce considerably better results using the same annotation time.

This paper describes an efficient system for scalable video summarization that exploits comic-like summaries and multi-scale representations to facilitate interactivity and balance between content coverage and compactness. Due to the layout disturbance induced by the transitions between scales, a new heuristic algorithm is proposed to restrict changes to bounded summary segments. Conducted user evaluations show that the proposed methodology improves usability while keeping the summaries compact and informative.

This letter presents an algorithm for detecting cross-fade scene changes in video carrying 3:2 or mixed film cadence. There are many existing methods for detecting gradual video transitions, but none of the current literature addresses the complication of film cadence. The differences between video and film capture and the mechanics of the telecine transfer process used to convert 24-Hz film to the main international television standards alter the temporal properties in a nonlinear way that makes cross-fades more difficult to detect with existing methods. An algorithm is proposed to address this problem.

This article proposes formal models for two commonly used methods - tagging and browsing - and investigates new approaches to improve the efficiency of manual image annotation.

This article has provided a brief overview of how the marriage of speech and language technologies with the Web is changing the way people communicate and access information. Advances are happening at a rapid pace through improved algorithms, wider penetration of the Web, availability of data, and faster computing. As the Web continues to evolve, research initiatives need to continue to address difficult challenges in areas of information mining of multimedia data, multimodal search of Web and media contents, speaker recognition for reducing Internet fraud, interactive question/answering for Web-based self-service, Web mining for knowledge discovery, two-way language translation, and Web page personalization and ranking. Progress in these areas will help to generate exciting new business opportunities for mobile Internet, secure voice print, ubiquitous multilingual communication, IPTV, globalization of customer care, and information search of massive amounts of data.

The rapidly increasing proliferation of digital images and videos leads to a situation where content-based search in multimedia databases becomes more and more important. A prerequisite for effective image and video search is to analyze and index media content automatically. Current approaches in the field of image and video retrieval focus on semantic concepts serving as an intermediate description to bridge the “semantic gap” between the data representation and the human interpretation. Due to the large complexity and variability in the appearance of visual concepts, the detection of arbitrary concepts represents a very challenging task. In this thesis, the following aspects of visual concept detection systems are addressed: First, enhanced local descriptors for mid-level feature coding are presented. Based on the observation that scale-invariant feature transform (SIFT) descriptors with different spatial extents yield large performance differences, a novel concept detection system is proposed that combines feature representations for different spatial extents using multiple kernel learning (MKL). A multi-modal video concept detection system is presented that relies on Bag-of-Words representations for visual and in particular for audio features. Furthermore, a method for the SIFT-based integration of color information, called color moment SIFT, is introduced. Comparative experimental results demonstrate the superior performance of the proposed systems on the Mediamill and on the VOC Challenge. Second, an approach is presented that systematically utilizes results of object detectors. Novel object-based features are generated based on object detection results using different pooling strategies. For videos, detection results are assembled to object sequences and a shot-based confidence score as well as further features, such as position, frame coverage or movement, are computed for each object class. These features are used as additional input for the support vector machine (SVM)-based concept classifiers. Thus, other related concepts can also profit from object-based features. Extensive experiments on the Mediamill, VOC and TRECVid Challenge show significant improvements in terms of retrieval performance not only for the object classes, but also in particular for a large number of indirectly related concepts. Moreover, it has been demonstrated that a few object-based features are beneficial for a large number of concept classes. On the VOC Challenge, the additional use of object-based features led to a superior performance for the image classification task of 63.8% mean average precision

The problem of content-based video retrieval continues to pose a challenge to the research community, the performance of video retrieval systems being low due to the semantic gap. In this paper we consider whether taking advantage of context can aid the video retrieval process by making the prediction of relevance easier, i.e. if it is easier for a classification system to predict the relevance of a video shot under a given context, then that context has potential in also improving retrieval, since the underlying features better differentiate relevant from non-relevant video shots. We use an operational definition of context, where datasets can be split into disjoint sub-collections which reflect a particular context. Contexts considered include task difficulty and user expertise, among others. In the classification process, four main types of features are used to represent video-shots: conventional low-level visual features representing physical properties of the video shots, behavioral features which are based on user interaction with the video shots, and two different bag-of-words features obtained from the Automatic Speech Recognition from the audio of the video. So, we measure how well each kind of video representation performs and, for each of these representations, our datasets are then split into different contexts in order to discover which contexts affect the performance of a number of trained classifiers. Thus, we aim to discover contexts which improve the classifier's performance and if this improvement is consistent regardless the kind of representation. Experimental results show which kind of the tested document representations works best for the different features; following on from this, we then identify the contexts which result in the classifiers performing better.

The problem of content-based video retrieval continues to pose a challenge to the research community, the performance of video retrieval systems being low due to the semantic gap. In this paper we consider whether taking advantage of context can aid the video retrieval process by making the prediction of relevance easier, i.e. if it is easier for a classication system to predict the relevance of a video shot under a given context, then that context has potential in also improving retrieval, since the underlying features better dierentiate relevant from non-relevant video shots. We use an operational denition

The problem of content-based image and video retrieval with textual queries is often posed as that of visual concept classification, where classifiers for a set of predetermined visual concepts are trained using a set of manually annotated images. Such a formulation implicitly assumes that the training data has similar distributional characteristics as that of the data which need to be indexed. In this paper we demonstrate empirically that even within the relatively narrow domain of news videos collected from a variety of news programs and broadcasters, the assumption of distributional similarity of visual features does not hold across programs from different broadcasters. This is manifested in considerable degradation of ranked retrieval performance on novel sources. We observe that concepts whose spatial locations remain relatively fixed between various sources are also more robust to source mismatches, and vice versa. We also show that a simple averaging of multiple visual detectors is more robust than any of the individual detectors. Furthermore, we show that for certain sources using only 20% of the available annotated data can bridge roughly 80% of the performance drop, while others can require larger amounts of annotated data.

The TREC Video Retrieval Evaluation (TRECVID) 2011 was a TREC-style video analysis and retrieval evaluation, the goal of which remains to promote progress in content-based exploitation of digital video via open, metrics-based evaluation. Over the last ten years this effort has yielded a better understanding of how systems can effectively accomplish such processing and how one can reliably benchmark their performance. TRECVID is funded by the National Institute of Standards and Technology (NIST) and other US government agencies. Many organizations and individuals worldwide contribute significant time and effort

The TREC Video Retrieval Evaluation (TRECVID) 2009 was a TREC-style video analysis and retrieval evaluation, the goal of which was to promote progress in content-based exploitation of digital video via open, metrics-based evaluation. Over the last 9 years TRECVID has yielded a better understanding of how systems can effectively accomplish such processing and how one can reliably benchmark their performance. 63 teams from various research organizations — 28 from Europe, 24 from Asia, 10 from North America, and 1 from Africa — completed one or more of four tasks: high-level feature extraction, search (fully automatic, manually assisted, or interactive), copy detection, or surveillance event detection. This paper gives an overview of the tasks, data used, evaluation mechanisms and performance

The TREC Video Retrieval Evaluation (TRECVID) 2007 represents the seventh running of a TREC-style (trec.nist.gov) video retrieval evaluation, the goal of which remains to promote progress in content-based retrieval from digital video via open, metrics-based evaluation. Over time this effort should yield a better understanding of how systems can effectively accomplish such retrieval and how one can reliably benchmark their performance. TRECVID 2007 was funded by the US National Institute of Standards and Technology (NIST) and the Intelligence Advanced Research Projects Activity (IARPA). 54 teams (see Table 1 at the end of the paper) from various research organizations — 17 from Asia, 23 from Europe, 12 from the Americas, and 2 from Australia — participated in one or more of four tasks: shot boundary determination, high-level feature extraction, search (fully automatic, manually assisted, or interactive) or pre-production video (rushes) summarization. See Figure 1 for an overview of TRECVID’s evolution. In 2007 TRECVID began what sets out to be a 3-year cycle using new data sources, related to the broadcast news used in 2003-2006 but significantly different. Data for the search and feature tasks was about 100 hours of (MPEG-1) news magazine, science news, news reports, documentaries, educational programming, and archival video almost entirely in Dutch from the Netherlands Institute for Sound and Vision. About 6 additional hours of Sound and Vision data was used for the shot boundary task. The BBC Archive provided about 50 hours of “rushes” pre-production video material with natural sound, errors, etc. from several BBC dramatic series for use in the summarization task.