With the advancement in computer vision deep learning, systems now are able to analyze an unprecedented amount of rich visual information from videos to enable applications such as autonomous driving, socially-aware robot assistant and public safety monitoring. Deciphering human behaviors to predict their future paths/trajectories and what they would do from videos is important in these applications. However, modern vision systems in self-driving applications usually perform the detection (perception) and prediction task in separate components, which leads to error propagation and sub-optimal performance. More importantly, these systems do not provide high-level semantic attributes to reason about pedestrian future. This design hinders prediction performance in video data from diverse domains and unseen scenarios. To enable optimal future human behavioral forecasting, it is crucial for the system to be able to detect and analyze human activities leading up to the prediction period, passing informative features to the subsequent prediction module for context understanding. In this thesis, with the goal of improving the performance and generalization ability of future trajectory and action prediction models, we conduct human action analysis and jointly optimize models for action detection, prediction and trajectory prediction.

We propose a video analysis system detecting activities in surveillance scenarios which wins Trecvid Activities in Extended Video (ActEV1) challenge 2019. For detecting and localizing surveillance events in videos, Argus employs a spatialtemporal activity proposal generation module facilitating object detection and tracking, followed by a sequential classification module to spatially and temporally localize persons and objects involved in the activity. We detail the design challenges and provide our insights and solutions in developing the state-of-the-art surveillance video analysis system.

We participated in one of the two types of Instance Search task in TRECVID 2019: Fully Automatic Search, without any human intervention. Firstly, the specific person and action are searched separately, and then we re-rank the two sorts of search results by ranking the one type scores according to the other type, as well as the score fusion. And thus, three kinds of final instance search results are submitted. Specifically, for the person search, our baseline consists of face detection, alignment and face feature selection. And for the action search, we integrate person detection, person tracking and feature selection into a framework to get the final 3D features for all tracklets in video shots. The official evaluations showed that our best search result gets the 4th place in the Automatic search.

Using a collection of publicly available links to short form video clips of an average of 6 seconds duration each, 1,275 users manually annotated each video multiple times to indicate both longterm and short-term memorability of the videos. The annotations were gathered as part of an online memory game and measured a participant’s ability to recall having seen the video previously when shown a collection of videos. The recognition tasks were performed on videos seen within the previous few minutes for short-term memorability and within the previous 24 to 72 hours for long-term memorability. Data includes the reaction times for each recognition of each video. Associated with each video are text descriptions (captions) as well as a collection of image-level features applied to 3 frames extracted from each video (start, middle and end). Video-level features are also provided. The dataset was used in the Video Memorability task as part of the MediaEval benchmark in 2020. Specifications Table ∗Corresponding author. e-mail: Alan.Smeaton@DCU.ie (Alan F. Smeaton) Preprint submitted to Data in BriefDecember 7, 2021 ar X iv :2 11 2. 02 30 3v 1 [ cs .C V ] 4 D ec 2 02 1 2 R.S. Kiziltepe et. al / Data in Brief (2021) Subject Computer Vision and Pattern Recognition Specific subject area Ground truth data (videos, video features plus annotations) needed to build and train systems for the automatic computation of the memorability of short video clips Type of data Text files (csv) How data were acquired Raw videos are already publicly available online. Low level features were extracted automatically from videos and annotation data was collected through crowdsourcing using a video memorability game with the participation of both volunteers and paid workers on Amazon Mechanical Turk. Data format Raw Analyzed Parameters for data collection The maximum false alarm rate (short-term): 30% The maximum false alarm rate (long-term): 40% The minimum recognition rate of vigilance fillers (short-term): 70% The minimum recognition rate (long-term): 15% The false alarm rate must be lower than the recognition rate (long-term). Description of data collection 1,500 short videos selected from the Vimeo Creative Commons (V3C1) dataset and used in the TRECVid 2019 Video-to-Text task were divided into three non-overlapping subsets: training, development, and testing. Multiple manual memorability annotations for each video were collected via a video memorability game, which displays a series of short videos and requires users to press the spacebar when they recall a video previously seen by them. The game consists of two parts: in the first part where videos are repeated within a few minutes, the user interaction with a repeated video was collected to calculate short-term memorability scores. The second part took place between 24 and 72 hours after initial viewing of videos, and this time the participants’ responses to previously seen videos from the first part were collected to acquire long-term memorability scores. After analysing the collected annotations, the short-term and the long-term memorability scores of each video were calculated as a percentage of correctly recalled videos, respectively. Each video memorabiity annotation is accompanied by the video timepoint offsets at which it was recalled by users, response times of the users, the key pressed when watching each video, and textual captions describing each video from the TRECVid benchmark. The Media Memorability 2020 dataset is included here with memorability annotations on 590 videos as part of the training set and 410 additional videos as part of the development set. In this dataset we provide memorability annotations for the development and training set videos but not the test set as this is used in future MediaEval memorability benchmark tasks. Data source location Primary data sources: TRECVid 2019 Video-to-Text dataset [1], available from: https: //www-nlpir.nist.gov/projects/trecvid/trecvid.data.html Institution: National Institute of Standards and Technology (NIST) City: Gaithersburg Country: USA Data accessibility Repository name: Figshare Direct URL to data: https://doi.org/10.6084/m9.figshare.15105867.v2 Source code used to process this is adapted from [2] and is available at https://github. com/InterDigitalInc/VideoMemAnnotationProtocol/ Related research article A. Garcı́a Seco de Herrera, R. Savran Kiziltepe, J. Chamberlain, M. G. Constantin, C.-H. Demarty, F. Doctor, B. Ionescu, A. F. Smeaton Overview of MediaEval 2020 Predicting Media Memorability Task: What Makes a Video Memorable? MediaEval Workshop, online 14-15 December, 2020 [3] DOI:

This year, the PicSOM and EURECOM teams participated only in the Video to Text Description (VTT), Description Generation subtask. Both groups submitted one or two runs labeled as a ”MeMAD” submission, stemming from a joint EU H2020 research project with that name. In total, the PicSOM team submitted four runs and EURECOM one run. The goal of the PicSOM submissions was to study the effect of using either image or video features or both. The goal of the EURECOM submission was to experiment with the use of Curriculum Learning in video captioning. The submitted five runs are as follows: • PICSOM.1-MEMAD.PRIMARY: uses ResNet and I3D features for initialising the LSTM generator, and is trained on MS COCO + TGIF using self-critical loss, • PICSOM.2-MEMAD: uses I3D features as initialisation, and is trained on TGIF using self-critical loss, • PICSOM.3: uses ResNet features as initialisation, and is trained on MS COCO + TGIF using self-critical loss, • PICSOM.4: is the same as PICSOM.1-MEMAD.PRIMARY except that the loss function used is cross-entropy, • EURECOM.MEMAD.PRIMARY: uses I3D features to initialize a GRU generator, and is trained on TGIF + MSR-VTT + MSVD with cross-entropy and curriculum learning. The runs aim at comparing the use of cross-entropy and self-critical training loss functions and to showing whether one can successfully use both still image and video features even when the COCO dataset does not allow the extractions of I3D video features. Based on the results of the runs, it seems that using both video and still image features, one can obtain better captioning results than with either one of the single modalities alone. The Curriculum Learning process proposed does not seem to be beneficial.

This report presents our system developed for Ad-hoc Video Search(AVS) task in TRECVID 2019 as Team ATL. In this AVS task, we apply a hybrid sequential encoder which make use of the utilities of not only the multi-modal sources but also the feature extractors such as GRU, aggregated vectors, graph modeling, etc. Our motivation is mapping video embedding and language embedding into a learned semantic space. We observe that by combining different models and make use of their utilities for feature extraction, we can take better advantage of large batches and hard examples. Our models are trained on MSRVTT [1], IACC.3,TGIF and TRECVID2016 VTT datasets with different hybrid visual and text architectures. The final ensemble model achieves 0.163 infap and won the first place in this task.

This paper provides an overview of the runs submitted to TRECVID 2018 by ITI-CERTH. ITI-CERTH participated in the Ad-hoc Video Search (AVS), Instance Search (INS) and Activities in Extended Video (ActEV) tasks. Our AVS task participation is based on a method that combines the linguistic analysis of the query with concept-based and semantic-embedding representations of video fragments. The INS task is performed by employing VERGE, which is an interactive retrieval application that integrates retrieval functionalities that consider mainly visual information. For the ActEV task, we deploy a novel activity detection algorithm that is based on human detection in video frames, goal descriptors, dense trajectories, Fisher vectors and a discriminative action segmentation scheme.

This paper presents our system developed for Adhoc Video Search (AVS) task in TRECVID 2019. Our system is based on embedding that maps visual and textual information into a common space to measure the relevance of each shot to a topic. We devise three embedding models built on two sources of training data, MS-COCO [1] and Flickr 30k [2]. Image feature extractors and region detector internally used in these models are pre-trained on ImageNet [3] and Visual Genome [4], respectively. The following five variants of our system were submitted: 1) F M C D kindai kobe.19 1: This run is an ensemble of three embedding models. The first and second models are respectively trained on MS-COCO and Flickr 30k to perform different coarse-grained embeddings between frames and a topic. The last model forms fine-grained embedding between regions in frames and words in a topic. 2) F M C D kindai kobe.19 2: This run is the same to F M C D kindai kobe.19 1 except that the fine-grained embedding model normalises regional features. 3) F M C D kindai kobe.19 3: This run only uses the finegrained embedding model without the normalisation. 4) F M C D kindai kobe.19 4: This run is an ensemble of only the two coarse-grained embedding models. 5) F M N D kindai kobe.19 5: This run is the same to F M C D kindai kobe.19 3 except the fine-grained embedding model using the normalisation. The MAPs of F M C D kindai kobe.19 3, F M C D kindai kobe.19 4 and F M N D kindai kobe.19 5 are 0.080, 0.059 and 0.081, respectively. This indicates that fine-grained embedding is much more effective than coarse-grained one. Considering that both F M C D kindai kobe.19 1’s and F M C D kindai kobe.19 2’s MAPs are 0.087, the ensemble of coarse-grained and fine-grained embeddings leads to small performance improvements, compared to F M C D kindai kobe.19 3 and F M C D kindai kobe.19 5. This means that the performances of the former are mainly owing to the latter. Finally, F M C D kindai kobe.19 1 and F M C D kindai kobe.19 2 are ranked at the fifth position in terms of teams participating in the fully automatic category, and our runs achieve the best MAPs for three topics in this category.

This paper presents details of a frame-based keyword search component that was used for TRECVID 2019 Ad-hoc Video Search with manually-assisted querying. The component is part of the VIRET framework primarily developed for interactive known-item search. For each task one query was formulated and the query was used to form four dierent runs by applying dierent scoring functions.

This paper presents a summary and results for the ActEV’20 SDL (Activities in Extended Video Sequestered Data Leaderboard) challenge that was held under the CVPR’20 ActivityNet workshop [38]. The primary goal of the challenge was to provide an impetus for advancing research and capabilities in the field of human activity detection in untrimmed multi-camera videos. Advancements in activity detection will help with a wide range of public safety applications. The challenge was administered by the National Institute of Standards and Technology (NIST), where anyone could submit their system which run on sequestered data with the resulting score posted to a public leaderboard. Ten teams submitted their systems for the ActEV’20 SDL competition on the Multiview Extended Video with Activities (MEVA) test set with 37 target activities. The performance metric for the leaderboard ranking is the partial, normalized Area Under the Detection Error Tradeoff (DET) curve (nAUDC). The top rank on activity detection was by UCF at 37%, followed by CMU at 39% and OPPO at 41%.

This paper describes the MediaEval 2021 Predicting Media Memorability task, which is in its 4th edition this year, as the prediction of short-term and long-term video memorability remains a challenging task. In 2021, two datasets of videos are used: first, a subset of the TRECVid 2019 Video-to-Text dataset; second, the Memento10K dataset in order to provide opportunities to explore cross-dataset generalisation. In addition, an Electroencephalography (EEG)-based prediction pilot subtask is introduced. In this paper,we outline themain aspects of the task and describe the datasets, evaluationmetrics, and requirements for participants’ submissions.

This paper describes the MediaEval 2020 Predicting Media Memorability task. After first being proposed at MediaEval 2018, the Predicting Media Memorability task is in its 3rd edition this year, as the prediction of short-term and long-term video memorability (VM) remains a challenging task. In 2020, the format remained the same as in previous editions. This year the videos are a subset of the TRECVid 2019 Video-to-Text dataset, containing more action rich video content as compared with the 2019 task. In this paper a description of some aspects of this task is provided, including its main characteristics, a description of the collection, the ground truth dataset, evaluation metrics and the requirements for participants’ run submissions.

This notebook reports the model and results of the EURECOM runs at TRECVID AVS 2019.

This notebook paper presents the system design of the FDU team in the TRECVID 2019 [1] VTT task. Our approach adopts temporal concept prediction as an auxiliary task to assist caption generation. The concept prediction module generates a context sequence that contains latent semantic features, which are later fused into the captioning module. We demonstrate the effectiveness of our designed auxiliary task as well as the whole captioning system.

This article conducts user evaluation to study the performance difference between interactive and automatic search. Particularly, the study aims to provide empirical insights of how the performance landscape of video search changes, with tens of thousands of concept detectors freely available to exploit for query formulation. We compare three types of search modes: free-to-play (i.e., search from scratch), non-free-to-play (i.e., search by inspecting results provided by automatic search), and automatic search including concept-free and concept-based retrieval paradigms. The study involves a total of 40 participants; each performs interactive search over 15 queries of various difficulty levels using two search modes on the IACC.3 dataset provided by TRECVid organizers. The study suggests that the performance of automatic search is still far behind interactive search. Furthermore, providing users with the result of automatic search for exploration does not show obvious advantage over asking users to search from scratch. The study also analyzes user behavior to reveal insights of how users compose queries, browse results, and discover new query terms for search, which can serve as guideline for future research of both interactive and automatic search.

The objective of this work is person-clustering in videos – grouping characters according to their identity. Previous methods focus on the narrower task of face-clustering, and for the most part ignore other cues such as the person’s voice, their overall appearance (hair, clothes, posture), and the editing structure of the videos. Similarly, most current datasets evaluate only the task of face-clustering, rather than person-clustering. This limits their applicability to downstream applications such as story understanding which require person-level, rather than only face-level, reasoning.In this paper we make contributions to address both these deficiencies: first, we introduce a Multi-Modal High-Precision Clustering algorithm for person-clustering in videos using cues from several modalities (face, body, and voice). Second, we introduce a Video Person-Clustering dataset, for evaluating multi-modal person-clustering. It contains body-tracks for each annotated character, face-tracks when visible, and voice-tracks when speaking, with their associated features. The dataset is by far the largest of its kind, and covers films and TV-shows representing a wide range of demographics. Finally, we show the effectiveness of using multiple modalities for person-clustering, explore the use of this new broad task for story understanding through character co-occurrences, and achieve a new state of the art on all available datasets for face and person-clustering.

The conventional method used entire frames which sampled at regular intervals from video clip to generate text descriptions [1]. Almost generating description methods are using deep learning techniques, such as encoder-decoder framework [2] [3]. Therefore, a learning and text generation take a long time if video has a lot of frames. In other words, processing time is increasing with the length of the video clip. There is a method using keyframe of video as conventional researches in order to reduce processing frames to generate explanation texts. But this method does not consider time step variation of video clips. Video is a set of consecutive images, but it is clearly different from simple image because video has information of time step. We assume that it is important for shortening process time that is not only reducing processing frames but also considering time steps in a video clip. Humans can create explanation texts only using some of frames not include whole of frames. In particular, we can recognize the content using only few frames if movie is short like which is uploaded on Vine and Twitter. For this reason, we propose a new method for reducing process frame with sustaining semantics and compared proposed method and conventional ones on score and processing time with doing evaluation experiment.

The analysis of video footage involving tasks such as identifying certain individuals at defined locations in a complex indoor or outdoor scenes or classifying person’s activities still poses a challenge to any video retrieval system. Nowadays a variety of (semi-)automated analysis systems can be applied in order to solve some of its specific subproblems and the accuracy of object detection and simultaneously the detection and classification accuracy benefits strongly when latest cutting-edge machine learning methods such as deeplearning networks are involved. In this paper we propose our design of a heterogeneous video analysis system and report about our experiences with its application to the Activity of Extended Video (ActEV) analysis task within the TREC Video Retrieval Evaluation (TRECVID) contest. The proposed system improves the performance of person detection, identification and localization at predefined places in video scenes by heuristically combining a variety of state-of-the-art deep-learning frameworks for object detection and places classification into one heterogeneous system. The incorporated frameworks adress a wide range of subproblems including stable object boundary extraction of salient regions or identifiable objects as well as person identification and object / place classification. Our approach integrates these processing artifacts using a feature-oriented approach in order to assess statistical correlations together with LSTM based activity classifiers across video frames.

The Waseda Meisei SoftBank team participated in the TRECVID 2019 Ad-hoc Video Search (AVS) task [1]. For this year’s AVS task, we submitted both manually assisted and fully automatic runs. Our approach consisted of concept-based video retrieval for manually assisted runs and visual-semantic embedding for fully automatic runs. Our best manually assisted run achieved a mean average precision (mAP) of 15.2%, which ranked the highest among all the manually assisted systems. Our fully automatic run achieved an mAP of 12.3%, which ranked third among all participants. 1 System Description We used two approaches for video retrieval from large-scale video data using query sentences. This section introduces how both systems were created. 1.1 Concept-based Approach For a concept-based approach, we first built a large concept bank comprising of several concept types as shown in Table 1. It contained classifiers such as persons, objects, scenes, and actions to deal with various forms of query sentences. Using this concept bank, all concept scores for all videos were calculated. Here, we explain how to create concept classifiers for each database and pre-trained models. 1. TRECVID346, FCVID239, UCF101, and ACTIVITYNET200 First, a maximum of ten frames from each shot were selected at regular intervals, and the corresponding images were input to the GoogLeNet model [11] pre-trained on the ImageNet database [6]. This allowed us to obtain 1,024-dimensional feature vectors from pool5 layers. These feature vectors (maximum ten) were then bound to a single vector using element-wise max-pooling. We trained SVMs using datagiven labels of concepts for each database as positive samples and randomly selected images from the TRECVID SIN dataset as negative samples. The shot score for each concept was calculated as the distance to the hyperplane in the SVM model.

The Predicting Media Memorability task in MediaEval’20 has some challenging aspects compared to previous years. In this paper we identify the high-dynamic content in videos and dataset of limited size as the core challenges for the task, we propose directions to overcome some of these challenges and we present our initial result in these directions.