Years of international participation and collaboration in TRECVID have shown that interactive multimedia systems, those with a user in the search loop, have consistently outperformed fully automated systems. Interface capabilities like querying by text, by image, and by semantic concept and storyboard layouts have led to significant performance improvements on provided search tasks. Lessons learned for TRECVID shot-based video retrieval are presented. In the real world, however, video collection users may focus on story threads instead of shots, or may not be provided with a clear stated search task. The paper also discusses users facing situations where they lack the knowledge or contextual awareness to formulate queries and having a genuine need for exploratory search systems supporting serendipitous browsing. Interfaces promoting various views for navigating complex information spaces can help with exploratory search and investigation into video corpora ranging from documentaries to broadcast news to oral histories. Work is presented using The HistoryMakers oral history archive as well as TRECVID international broadcast news to discuss the utility of various query and presentation mechanisms emphasizing people, time, location, and visual attributes. The paper leads into a discussion of how exploratory interfaces for video extend beyond storyboards, with a series of user studies referenced as empirical data in support of the presented conclusions.

With an increasing amount of video data in our daily life, the need for content-based search in videos increases as well. Though a lot of research has been spent on video retrieval tools and methods which allow for automatic search in videos through content-based queries, still the performance of automatic video retrieval is far from optimal. In this tutorial we discussed (i) proposed solutions for improved video content navigation, (ii) typical interaction of content-based querying features, and (iii) advanced video content visualization methods. Moreover, we discussed interactive video search systems and ways to evaluate their performance.

We propose a novel video integration architecture, INTERVIDEO, for faceted search on web-scale. First, we demonstrate that the traditional video integration techniques are no longer valid in face of such heterogeneity and scale. Then, we present three new integrating techniques to build a global relation schema for organizing web videos and aiding user to retrieve faceted results. Finally, we conduct an experimental study and demonstrate the ability of our system to automatically integrate videos and build a complete and concise high-level relation schema on large, heterogeneous web sites.

We have developed an interactive video search system that allows the searcher to rapidly assess query results and easily pivot off those results to form new queries. The system is intended to maximize the use of the discriminative power of the human searcher. This is accomplished by providing a hierarchical segmentation, streamlined interface, and redundant visual cues throughout. The typical search scenario includes a single searcher with the ability to search with text and content-based queries. In this paper, we evaluate new variations on our basic search system. In particular we test the system using only visual content-based search capabilities, and using paired searchers in a realtime collaboration. We present analysis and conclusions from these experiments.

We have developed an interactive video search system that allows the searcher to rapidly assess query results and easily pivot o those results to form new queries. The system is intended to maximize the use of the discriminative power of the human searcher. The typical video search scenario we consider has a single searcher with the ability to search with text and content-based queries. In this paper, we evaluate a new collaborative modication of our search system. Using our system, two or more users with a common information need search together, simultaneously. The collaborative system provides tools, user interfaces and, most importantly, algorithmically-mediated retrieval to focus, enhance and augment the team’s search and communication activities. In our evaluations, algorithmic mediation improved the collaborative performance of both retrieval (allowing a team of searchers to nd relevant information more eciently and eectively),

We describe StartMobile, a prototype system that enables users of cellular telephones and other mobile devices to access information, create information and execute commands on the basis of written requests expressed in natural language. Interacting with mobile devices through the use of language offers several potential benefits that include: scalability in handling huge numbers of concepts and operations, substantial device-independence, and opportunities for remote interaction. StartMobile uses the START information access system to perform initial interpretation of requests and to generate responses to general information requests. Requests that require further interpretation and/or fulfillment in the distributed mobile environment are encoded in a new, language-based intermediate representation called Moebius, then conveyed to appropriate devices and systems, where special-purpose software serves to complete the interpretation and fulfillment of those requests.

Video surveillance is a key tool for enabling security personnel to monitor complex and dangerous environments, and the access to the semantic content of surveillance video has become a challenging research area. In this paper we present a query-processing system for investigating human activities in surveillance videos. The interaction between the user and the system happens through a gesture-based dialogue, which implements a novel question-answering model to support users during the information-seeking process. We also present a visual interface devised to facilitate the query-specification

Various query methods for video search exist. Because of the semantic gap each method has its limitations. We argue that for effective retrieval query methods need to be combined at retrieval time. However, switching query methods often involves a change in query and browsing interface, which puts a heavy burden on the user. In this paper, we propose a novel method for fast and effective search trough large video collections by embedding multiple query methods into a single browsing environment. To that end we introduced the notion of query threads, which contain a shot-based ranking of the video collection according to some feature-based similarity measure. On top of these threads we define several thread-based visualizations, ranging from fast targeted search to very broad exploratory search, with the Fork-Browser as the balance between fast search and video space exploration. We compare the effectiveness and efficiency of the ForkBrowser with the CrossBrowser on the TRECVID 2007 interactive search task. Results show that different query methods are needed for different types of search topics, and that the ForkBrowser requires significantly less user interactions to achieve the same result as the CrossBrowser. In addition, both browsers rank among the best interactive retrieval systems currently available.

This paper introduces an effective interactive video retrieval system named VisionGo. It jointly explores human and computer to accomplish video retrieval with high effectiveness and efficiency. It assists the interactive video retrieval process in different aspects: (1) it maximizes the interaction efficiency between human and computer by providing a user interface that supports highly effective user annotation and an intuitive visualization of retrieval results; (2) it employs a multiple feedback technique that assists users in choosing proper method to enhance relevance feedback performance; and (3) it facilitates users to assess the retrieval results of motion-related queries by using motion-icons instead of static keyframes. Experimental results based on over 160h of news video shows demonstrate the effectiveness of the VisionGo system.

This paper describes a multimedia multimodal information access sub-system (MIAS) for digital audio-visual documents, typically presented in streaming media format. The system is designed to provide both professional and general users with entry points into video documents that are relevant to their information needs. In this work, we focus on the information needs of multimedia specialists at a Dutch cultural heritage institution with a large multimedia archive. A quantitative and qualitative assessment is made of the efficiency of search operations using our multimodal system and it is demonstrated that MIAS significantly facilitates information retrieval operations when searching within a video document.

This is an exploratory study investigating the concept of combining social and automated digital video surrogates with the intention of improving user understanding and retrieval of video. Videos were shown to participants who were then asked to tag the videos according to two systems. The control group tagged videos in the traditional manner of tagging the whole video in one input form. The experimental group tagged a storyboard with multiple input forms corresponding to the storyboard. The findings of this study suggest that the implementation of a system that combines automatically generated storyboards with social tagging will significantly increase the number of tags that each user will add to a video. In all aspects of the results, there was significant increase in the amount of tags created in the storyboard system when compared to the traditional system of tagging an entire video. The results suggest that it is worth investigating in future work.

There are large amounts of digital video available. High recall retrieval of these requires going beyond the ranked results, which is the common target in high precision retrieval. To aid high recall retrieval, we propose Active Bucket Categorization, which is a multicategory interactive learning strategy which extends MediaTable , our multimedia categorization tool. MediaTable allows users to place video shots into buckets: user-assigned subsets of the collection. Our Active Bucket Categorization approach augments this by unobtrusively expanding these buckets with related footage from the whole collection. In this paper, we propose an architecture for active bucket-based video retrieval, evaluate two different learning strategies, and show its use in video retrieval with an evaluation using three groups of nonexpert users. One baseline group uses only the categorization features of MediaTable such as sorting and filtering on concepts and fast grid preview, but no online learning mechanisms. One group uses on-demand passive buckets. The last group uses fully automatic active buckets which autonomously add content to buckets. Results indicate a significant increase in the number of relevant items found for the two groups of users using bucket expansions, yielding the best results with fully automatic bucket expansions, thereby aiding high recall video retrieval significantly.

The last decade has seen innovations that make video recording, manipulation, storage, and sharing easier than ever before, thus impacting many areas of life. New video retrieval scenarios emerged as well, which challenge the state-of-the-art video retrieval approaches. Despite recent advances in content analysis, video retrieval can still benefit from involving the human user in the loop. We present our experience with a class of interactive video retrieval scenarios and our methodology to stimulate the evolution of new interactive video retrieval approaches. More specifically, the video browser showdown evaluation campaign is thoroughly analyzed, focusing on the years 2015–2017. Evaluation scenarios, objectives, and metrics are presented, complemented by the results of the annual evaluations. The results reveal promising interactive video retrieval techniques adopted by the most successful tools and confirm assumptions about the different complexity of various types of interactive retrieval scenarios. A comparison of the interactive retrieval tools with automatic approaches (including fully automatic and manual query formulation) participating in the TRECVID 2016 ad hoc video search task is discussed. Finally, based on the results of data analysis, a substantial revision of the evaluation methodology for the following years of the video browser showdown is provided.

The Carnegie Mellon University Informedia group has enjoyed consistent success with TRECVID interactive search using traditional storyboard interfaces for shot-based retrieval. For TRECVID 2006 the output of automatic search was included for the first time with storyboards, as an option for an interactive user. This access mechanism, query-by-best-of-topic, complements other access strategies, namely query-by-text (dominated by closed captioned text), query-by-image (primarily color-based syntactic lookup), and query-by-concept (using concept classifiers trained through machine learning techniques). The interface is tuned to support efficient performance required for TRECVID interactive search tasks where a maximum of 15 minutes can be spent per task. This 4-strategy Informedia interface scored significantly better than all other TRECVID 2006 interactive search systems. Interface attributes will be demonstrated in shot-based retrieval from news corpora.

Storyboards, a grid layout of thumbnail images as surrogates representing video, have received much attention in video retrieval interfaces and published studies through the years, and work quite well as navigation aids and as facilitators for shot-based information retrieval. When the information need is tied less to shots and requires inspection of stories and across stories, other interfaces into the video data have been demonstrated to be quite useful. These interfaces include scatterplots for timelines, choropleth maps, dynamic query preview histograms, and named entity relation diagrams representing sets of hundreds or thousands of video stories. One challenge for interactive video search is to move beyond support for fact-finding and also address broader, longer term search activities of learning, analysis, synthesis, and discovery. Examples are shown for broadcast news and life oral histories, drawing from empirically collected data showing how such interfaces can promote improved exploratory search. This paper surveys and reflects on a body of Informedia interface work dealing with news, folding in for the first time an examination of exploratory transactions with an oral history corpus.

Speech media, that is, digital audio and video containing spoken content, has blossomed in recent years. Large collections are accruing on the Internet as well as in private and enterprise settings. This growth has motivated extensive research on techniques and technologies that facilitate reliable indexing and retrieval. Spoken content retrieval (SCR) requires the combination of audio and speech processing technologies with methods from information retrieval (IR). SCR research initially investigated planned speech structured in document-like units, but has subsequently shifted focus to more informal spoken content produced spontaneously, outside of the studio and in conversational settings. This survey provides an overview of the field of SCR encompassing component technologies, the relationship of SCR to text IR and automatic speech recognition and user interaction issues. It is aimed at researchers with backgrounds in speech technology or IR who are seeking deeper insight on how these fields are integrated to support research and development, thus addressing the core challenges of SCR.

Semantic search service is generally considered to include text search, audio search, and video search. The annotation of a video is crucial to the success of video search and retrieval. With the precise annotation of video files and their semantic reasoning, more accurate search results according to the users’ query intention can be obtained. In this paper, a prototype of such a video annotation system, called Semantic Video Annotation System (SVAS), is presented. SVAS uses a three-level annotation architecture and a semantic video search language called Semantic Query Description Language for Video (SQDL-V). SQDL-V engine based on SVAS is able to return more accurate search results in comparison to the formal video search method.

Recorded interviews form a rich basis for scholarly inquiry. Examples include oral histories, community memory projects, and interviews conducted for broadcast media. Emerging technologies offer the potential to radically transform the way in which recorded interviews are made accessible, but this vision will demand substantial investments from a broad range of research communities. This article reviews the present state of practice for making recorded interviews available and the state-of-the-art for key component technologies. A large number of important research issues are identified, and from that set of issues, a coherent research agenda is proposed.

Ranked shot lists from 39 automated LSCOM-Lite concept classifiers are investigated with respect to 24 TRECVID 2006 topics. Selecting the best fitting concept or pair of concepts produces the shot set with greatest utility, rather than drawing fewer shots from a larger set of concepts. Mean average precision measures show concept-based shot sets have great utility for topics when perfectly traversed by a user. Using empirical data, however, shows that realistic ability to separate relevant shots from irrelevant ones and recall all the relevant ones is topic-dependent and far from perfect. Concept-based strategies including user-driven selection strategies not using idealized oracle prioritization are also discussed, with implications for query-by-concept in interactive video retrieval as concept spaces grow from tens to thousands.

Ranked shot lists from 39 automated LSCOM-Lite concept classifiers are investigated with respect to 24 TRECVID 2006 topics. Selecting the best fitting concept or pair of concepts produces the shot set with greatest utility, rather than drawing fewer shots from a larger set of concepts. Mean average precision measures show concept-based shot sets have great utility for topics when perfectly traversed by a user. Using empirical data, however, shows that realistic ability to separate relevant shots from irrelevant ones and recall all the relevant ones is topic-dependent and far from perfect. Concept-based strategies including user-driven selection strategies not using idealized oracle prioritization are also discussed, with implications for query-by-concept in interactive video retrieval as concept spaces grow from tens to thousands.