With the widespread use of smartphones as recording devices and the massive growth in bandwidth, the number and volume of video collections has increased significantly in the last years. This poses novel challenges to the management of these large-scale video data and especially to the analysis of and retrieval from such video collections. At the same time, existing video datasets used for research and experimentation are either not large enough to represent current collections or do not reflect the properties of video commonly found on the Internet in terms of content, length, or resolution.

With the recent resurgence of neural networks and the proliferation of massive amounts of unlabeled multimodal data, recommendation systems and multimodal retrieval systems based on continuous representation spaces and deep learning methods are becoming of great interest. Multimodal representations are typically obtained with autoencoders that reconstruct multimodal data. In this article, we describe an alternative method to perform high-level multimodal fusion that leverages crossmodal translation by means of symmetrical encoders cast into a bidirectional deep neural network (BiDNN). Using the lessons learned from multimodal retrieval, we present a BiDNN-based system that performs video hyperlinking and recommends interesting video segments to a viewer. Results established using TRECVIDs 2016 video hyperlinking benchmarking initiative show that our method obtained the best score, thus defining the state of the art.

With our submission to the 2017 TRECVID Instance Search task (Awad et al., 2017b), we focused on the usage of dedicated CNN models. We limited the available video training data to only three sources: The EastEnders episode 0, the given location examples and the given person examples. Our main workflow consists of three steps: First, we identify and crop persons from the training videos. Secondly, we train person and location classifiers using CNNs. Finally, we apply an ensemble strategy with prediction pooling to find matches for the given topics. In this contribution, we will provide some insights into our strategies and discuss results. Additionally, we present a novel way of interactive result annotation using HTC Vive VR headsets. 1 Structured Abstract 1. Briefly, list all the different sources of training data used in the creation of your system and its components. • We used the given master shot reference, the first episode with ID 0 (also denoted as DEV0 in this contribution) from the provided BBC EastEnders video footage as well as the location and person video examples. Additionally, we used textual metadata crawled from the BBC website containing episode descriptions. No other external training data was used. 2. Briefly, what approach or combination of approaches did you test in each of your submitted runs? • F E TUC HSMW 1: Dedicated model ensemble for person and location classification. The results of this run are re-ranked by similarity group scores. • F E TUC HSMW 2: Dedicated model ensemble like our first run, no re-ranking. • F E TUC HSMW 3: Faster-RCNN for person detection and classification. This is our best system from 2016. • I E TUC HSMW 4: Our only interactive run. Result re-ranking of our first run. We used a novel VR environment for the annotation process. Correspondence to: Stefan Kahl stefan.kahl@informatik.tu-chemnitz.de 3. What if any significant differences (in terms of what measures) did you find among the runs? • In contrast to last year’s submission, similarity group scores did not benefit the results. • We managed to retrieve 20% more relevant shots compared to our 2016 system. • Interactive re-ranking of results did not boost the overall scores as much as we expected. 4. Based on the results, can you estimate the relative contribution of each component of your system/approach to its effectiveness? • Person and location detection can be done using small, dedicated datasets for a closed domain environment like the EastEnders universe. • Simple CNN architectures are easy to train on small data sets but lack high generalization performance. • CNN ensembles perform significantly better than single models. • Interactive VR environments are suitable for interactive result-re-ranking but require more time than traditional annotation tools. 5. Overall, what did you learn about runs/approaches and the research question(s) that motivated them? 2 Kahl et. al.: TUC HSMW at TRECVID Instance Search 2017 • Increasing the generalization performance on small training sets is very challenging and requires extensive hyper-parameter tuning. • Different learning strategies aside from classification tasks might improve the performance (e.g. matching images of persons for similarities). • Ensemble strategies are strong and render some other techniques (e.g. similarity clustering) obsolete. The remainder of the paper is organized as follows: First, we provide a short workflow overview in section 2. After that, section 3 contains some insights into our training dataset. In section 4 we give a summary of our training process using artificial neural networks. Section 5 presents a novel approach for interactive evaluation on the EastEnders dataset. Finally, in section 6 we discuss the results of our submission.

Web image analysis has witnessed an AI renaissance. The ILSVRC benchmark has been instrumental in providing a corpus and standardized evaluation. The NVIDIA AI City Challenge is envisioned to provide similar impetus to the analysis of image and video data that helps make cities smarter and safer. In its first year, this Challenge has focused on traffic video data. While millions of traffic video cameras around the world capture data, albeit low-quality, very little automated analysis and value creation results. Lack of labeled data, and trained models that can be deployed at the edge of the city fabric, ensure that most traffic video data goes through little or no automated analysis. Real-time and batch analysis of this data can provide vital breakthroughs in real-time traffic management as well as pedestrian safety. The NVIDIA AI City Challenge brought together 29 teams from universities in 4 continents to collaboratively annotate a 125 hour data set and then compete on detection, localization and classification tasks as well as traffic and safety application analytics tasks. The result is the largest high quality annotated data set, a set of models trained using NVIDIA AI City Edge to Cloud platform and ready to be deployed at the edge solving traffic and safety problems for cities worldwide.

We report on our system used in the TRECVID 2017 Multimedia Event Detection (MED) and Ad-hoc Video Search (AVS) tasks. On the MED task, the CMU team submitted runs in 010Ex settings for the Pre-specified and Ad-hoc Events. On the AVS task, the CMU team submitted runs for fully-automatic system with no annotation condition.

We present a tutorial focusing on video retrieval tasks, where state-of-the-art deep learning approaches still benefit from interactive decisions of users. The tutorial covers general introduction to the interactive video retrieval research area, state-of-the-art video retrieval systems, evaluation campaigns and recently observed results. Moreover, a significant part of the tutorial is dedicated to a practical exercise with three selected state-of-the-art systems in the form of an interactive video retrieval competition. Participants of this tutorial will gain a practical experience and also a general insight of the interactive video retrieval topic, which is a good start to focus their research on unsolved challenges in this area.

We present a modern interactive video retrieval tool, called diveXplore, that has been used for several iterations of the Video Browser Showdown (VBS) competition with great success – 2nd place for the last two years in a row. The tool provides novel video content search and interaction features (e.g., a semantic map-search & browsing feature with similarity arrangement and a highly efficient sketch-search, optimized for mobile touch-interaction) that make it perfectly suited for flexible video retrieval in large video collections. With the help of a user study we show that the diveXplore system can be used very efficiently by both type of users: novices and experts. Our evaluation results do also show that the interaction statistics of novices and experts differ in terms of used features. The details of our insights can be used to further optimize interfaces of video retrieval tools for non-experts.

We present Direct Assessment, a method for manually assessing the quality of automatically-generated captions for video. Evaluating the accuracy of video captions is particularly difficult because for any given video clip there is no definitive ground truth or correct answer against which to measure. Metrics for comparing automatic video captions against a manual caption such as BLEU and METEOR, drawn from techniques used in evaluating machine translation, were used in the TRECVid video captioning task in 2016 but these are shown to have weaknesses. The work presented here brings human assessment into the evaluation by crowd sourcing how well a caption describes a video. We automatically degrade the quality of some sample captions which are assessed manually and from this we are able to rate the quality of the human assessors, a factor we take into account in the evaluation. Using data from the TRECVid video-to-text task in 2016, we show how our direct assessment method is replicable and robust and scales to where there are many caption-generation techniques to be evaluated including the TRECVid video-to-text task in 2017.

We participated in the matching and ranking subtask in TRECVid challenge 2017. The task here was to return a ranked list of the most likely text descriptions that correspond to each video. We adopted a joint visual semantic embedding approach for image-text retrieval and applied to the video-text retrieval task utilizing key-frames extracted by dissimilaritybased sparse subset selection approach. We trained our system on the MS-COCO dataset and tested on the TRECVid dataset. Our approach got an average mean inverted ranking score of 0.255 across 4 sets of testing data, and we ranked the 3rd overall in the challenge on this task.

We describe our approaches that were tested in the TRECVID 2017 Instance Search (INS) task. In this year’s INS, shots including a person at a location were to be retrieved after a location query and person query were given. We used ASMK for the location search and OpenFace for the person search. The results of the location search and person search were fused according to the ranks in the two searches or with the weighted sum of the zscores of these searches. We also applied human region removal to the sample images for the location search. We submitted four runs as follows for task A in which no video and only the provided sample images of the queries were used. NTT NII A run 1 : with human region removal and fusion with rank NTT NII A run 2 : without human region removal and fusion with rank NTT NII A run 3 : with human region removal and fusion with weighted sum of z-scores NTT NII A run 4 : with human region removal and fusion with weighted sum of z-scores 1 Our approach to Instance Search task The Instance Search task in TRECVID 2017 was to retrieve for each topic up to 1000 shots most likely to contain a query person at a query location, given a collection of test videos, a master shot reference, example images/videos of location queries, and example images/videos of person queries [1]. This year, we used no example video. We used only given example images; i.e., we tried task A only. For each topic, we retrieved shots containing the location given in the topic with the given example images for that location, and retrieved shots containing the person given in the topic with the given example images for this person; then, we integrated these retrieved shots and selected those containing both the location and the person. We tried only fully automatic search.

Video hyperlinking is a task aiming to enhance the accessibility of large archives, by establishing links between fragments of videos. The links model the aboutness between fragments for efficient traversal of video content. This paper addresses the problem of link construction from the perspective of cross-modal embedding. To this end, a generalized multi-modal auto-encoder is proposed. The encoder learns two embeddings from visual and speech modalities, respectively, whereas each of the embeddings performs self-modal and cross-modal translation of modalities. Furthermore, to preserve the neighbourhood structure of fragments, which is important for video hyperlinking, the auto-encoder is devised to model data distribution of fragments in a dataset. Experiments are conducted on Blip10000 dataset using the anchor fragments provided by TRECVid Video Hyperlinking (LNK) task over the years of 2016 and 2017. This paper shares the empirical insights on a number of issues in cross-modal learning, including the preservation of neighbourhood structure in embedding, model fine-tuning and issue of missing modality, for video hyperlinking.

Video description is the automatic generation of natural language sentences that describe the contents of a given video. It has applications in human-robot interaction, helping the visually impaired and video subtitling. The past few years have seen a surge of research in this area due to the unprecedented success of deep learning in computer vision and natural language processing. Numerous methods, datasets, and evaluation metrics have been proposed in the literature, calling the need for a comprehensive survey to focus research efforts in this flourishing new direction. This article fills the gap by surveying the state-of-the-art approaches with a focus on deep learning models; comparing benchmark datasets in terms of their domains, number of classes, and repository size; and identifying the pros and cons of various evaluation metrics, such as SPICE, CIDEr, ROUGE, BLEU, METEOR, and WMD. Classical video description approaches combined subject, object, and verb detection with template-based language models to generate sentences. However, the release of large datasets revealed that these methods cannot cope with the diversity in unconstrained open domain videos. Classical approaches were followed by a very short era of statistical methods that were soon replaced with deep learning, the current state-of-the-art in video description. Our survey shows that despite the fast-paced developments, video description research is still in its infancy due to the following reasons: Analysis of video description models is challenging, because it is difficult to ascertain the contributions towards accuracy or errors of the visual features and the adopted language model in the final description. Existing datasets neither contain adequate visual diversity nor complexity of linguistic structures. Finally, current evaluation metrics fall short of measuring the agreement between machine-generated descriptions with that of humans. We conclude our survey by listing promising future research directions.

Video content has been increasing at an unprecedented rate in recent years, bringing the need for improved tools providing efficient access to specific contents of interest. Within the management of video content, hyperlinking aims at determining related video segments from a collection with respect to an input video anchor. This paper describes the system we designed to address feature selection for the video hyperlinking challenge, as defined by TRECVID, one of the top worldwide venues for multimedia benchmarking. The proposed solution is based on different combinations of textual and visual features, enriched to capture the various facets of the videos: automatically generated transcripts, visual concepts, video metadata, named-entity recognition, and concept-mapping techniques. The different combinations of monomodal queries are experimentally evaluated, and the impact of both parameters and single features are discussed to identify their contributions. The best performing approach at the TRECVID 2017 video hyperlinking challenge was the ensemble feature selection, which includes three different monomodal queries based on enriched feature sets.

This work summarizes the findings of the 7th iteration of the Video Browser Showdown (VBS) competition organized as a workshop at the 24th International Conference on Multimedia Modeling in Bangkok. The competition focuses on video retrieval scenarios in which the searched scenes were either previously observed or described by another person (i.e., an example shot is not available). During the event, nine teams competed with their video retrieval tools in providing access to a shared video collection with 600 hours of video content. Evaluation objectives, rules, scoring, tasks, and all participating tools are described in the article. In addition, we provide some insights into how the different teams interacted with their video browsers, which was made possible by a novel interaction logging mechanism introduced for this iteration of the VBS. The results collected at the VBS evaluation server confirm that searching for one particular scene in the collection when given a limited time is still a challenging task for many of the approaches that were showcased during the event. Given only a short textual description, finding the correct scene is even harder. In ad hoc search with multiple relevant scenes, the tools were mostly able to find at least one scene, whereas recall was the issue for many teams. The logs also reveal that even though recent exciting advances in machine learning narrow the classical semantic gap problem, user-centric interfaces are still required to mediate access to specific content. Finally, open challenges and lessons learned are presented for future VBS events.

This short paper provides further details of the diveXplore system (formerly known as CoViSS), which has been used by team ITEC1 for the Video Browser Showdown (VBS) 2018. In particular, it gives a short overview of search features and some details of final system changes, not included in the corresponding VBS2018 paper, as well as a basic analysis of how the system has been used for VBS2018 (from a user perspective).

This short paper provides further details of the Sloth Search System, which was developed by the NECTEC team for the Video Browser Showdown (VBS) 2018.

This paper provides an overview of the runs submitted to TRECVID 2018 by ITI-CERTH. ITI-CERTH participated in the Ad-hoc Video Search (AVS), Instance Search (INS) and Activities in Extended Video (ActEV) tasks. Our AVS task participation is based on a method that combines the linguistic analysis of the query with concept-based and semantic-embedding representations of video fragments. The INS task is performed by employing VERGE, which is an interactive retrieval application that integrates retrieval functionalities that consider mainly visual information. For the ActEV task, we deploy a novel activity detection algorithm that is based on human detection in video frames, goal descriptors, dense trajectories, Fisher vectors and a discriminative action segmentation scheme.

This paper proposes a fundamental answer to a frequently asked question in multimedia computing and machine learning: Do artifacts from perceptual compression contribute to error in the machine learning process and if so, how much? Our approach to the problem is a reinterpretation of the Helmholtz Free Energy formula from physics to explain the relationship between content and noise when using sensors (such as cameras or microphones) to capture multimedia data. The reinterpretation allows a bit-measurement of the noise contained in images, audio, and video by combining a classifier with perceptual compression, such as JPEG or MP3. Our experiments on CIFAR-10 as well as Fraunhofer's IDMT-SMT-Audio-Effects dataset indicate that, at the right quality level, perceptual compression is actually not harmful but contributes to a significant reduction of complexity of the machine learning process. That is, our noise quantification method can be used to speed up the training of deep learning classifiers significantly while maintaining, or sometimes even improving, overall classification accuracy. Moreover, our results provide insights into the reasons for the success of deep learning.

This paper presents the runs that were submitted to the TRECVid Challenge 2017 for the Video Hyperlinking task. The goal of the task is to propose a list of video segments, called targets, to complement query video segments defined as anchors. The data provided with the task encourage participants to make use of multiple modalities such as the audio track and the keyframes. In this context, we submitted four runs: 1) BiDNNFull uses a BiDNN model to combine ResNet with Word2Vec; 2) BiDNNFilter makes use of the same model and also exploits the metadata to narrow down the list of possible candidates; 3) BiDNNPinv tries to improve on the anchor keyframe fusion by using the Moore-Penrose pseudo-inverse and finally 4) noBiDNNPinv tests on the relevance of not using a BiDNN to fuse the modalities. Our runs were built based on a pre-trained model of ResNet as well as the transcripts and the metadata provided by the organizers of the task. The results show a gain in performance over the baseline BiDNN model both when the metadata filter was used and when the keyframe fusion was done with a pseudo-inverse.

This paper presents the result of the TRECVID 2017 AVS task by kobe nict siegen team. Consisting of three research institutes Kobe University, NICT and University of Siegen. We submitted the following three runs. 1) kobe nict siegen D M 1: This run uses feature vectors extracted by a pre-trained convolutional neural network (CNN) as input for a small-scale multi-layer neural network called micro neural network (microNN). The microNN is a lightweight detector fine-tuned to a target concept using a balanced set of positive and negative data from ImageNet and IACC video datasets. Finally, the results of several microNNs are combined to generate the ranked result. 2) kobe nict siegen D M 2: This run is basically identical to kobe nict siegen D M 1, but applys additional motion features. The motion features are extracted by a motion CNN which involves biologically-inspired motion thresholding and competitive learning. 3) kobe nict siegen D M 3: This run is a degraded version of kobe nict siegen D M 1, and apply fully averaged pooling to feature vectors extracted with CNN, the dimention of the vector is reduced from 3× 3× 2048 to 2048. We also use a word2vec model to find synonyms of concepts in order to improve the performance.