NTU ROSE Lab at TRECVID 2018: Ad-hoc Video Search and Video to Text

This paper describes our participation in the ad-hoc video search and video to text tasks of TRECVID 2018. In ad-hoc video search, we adapted an image-based visual semantic embedding approach and trained our model on combined MS COCO and Flicker30k datasets. We extracted multiple keyframes from each shot and performed similarity search using the computed embeddings. In video to text, description generation task, we trained a video captioning model with multiple features using a reinforcement learning method on the combination of MSR-VTT and MSVD video captioning datasets. For the matching and ranking subtask, we trained two types of image-based ranking models on the MS COCO dataset. 1 Ad-hoc Video Search (AVS) In the ad-hoc video search task, we are given 30 free text queries and required to return the top 1000 shots from the test set videos [1, 2]. The queries are given in Appendix A. The test set contains 4593 Internet Archive videos of 600 hours with 450K shots (publicly available on TRECVID website). Videos have a duration of 6.5 minutes to 9.5 minutes. The reference shot boundaries are publicly available. No annotated training data was provided specifically for the AVS task. We participated with a “fully automatic” (Type F) system trained on already available annotated datasets (Type D), MS COCO and Flicker30k [5, 6]. 1.1 Visual Semantic Embedding We adapted the visual semantic embedding, VSE/VSE++ [3, 4], for cross modal retrieval. Given a set of image-caption pairs, VSE++ learns a joint embedding space, which can be used for cross-modal retrieval, i.e., given text, retrieve images/videos or vice versa.