Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

Employing large-scale pre-trained model CLIP to conduct video-text retrieval task (VTR) has become a new trend, which exceeds previous VTR methods. Though, due to the heterogeneity of structures and contents between video and text, previous CLIP-based models are prone to overfitting in the training phase, resulting in relatively poor retrieval performance. In this paper, we propose a multi-stream Corpus Alignment network with single gate Mixture-of-Experts (CAMoE) and a novel Dual Softmax Loss (DSL) to solve the two heterogeneity. The CAMoE employs Mixture-ofExperts (MoE) to extract multi-perspective video representations, including action, entity, scene, etc., then align them with the corresponding part of the text. In this stage, we conduct massive explorations towards the feature extraction module and feature alignment module, and conclude an efficient VTR framework. DSL is proposed to avoid the oneway optimum-match which occurs in previous contrastive methods. Introducing the intrinsic prior of each pair in a batch, DSL serves as a reviser to correct the similarity matrix and achieves the dual optimal match. DSL is easy to implement with only one-line code but improves significantly. The results show that the proposed CAMoE and DSL are of strong efficiency, and each of them is capable of achieving State-of-The-Art (SOTA) individually on various benchmarks such as MSR-VTT, MSVD, and LSMDC. Further, with both of them, the performance is advanced to a great extent, surpassing the previous SOTA methods for around 4.6% R@1 in MSR-VTT. The code will be available soon at https://github.com/starmemda/CAMoE/ Introduction Motivation The primary issue limiting VTR task presently is the heterogeneity between different modals, reflected in both structures and contents. The heterogeneity of structures. This mainly lies in the impossibility of directly aligning the words in sentences with corresponding video frames (Jin et al. 2021). Singlestream or two-stream structures are applied to treat text and video as two independent parts for early or late fusion, which *Interns at MMU, KuaiShou Inc. Corresponding author Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. ignore the internal relevancy between frames and words, resulting in that the models require massive data to reach decent performance. In this paper, we assume that texts can be parsed into separate sentences with distinct aspects of information. Though directly aligning a word with a frame is unachievable, guiding the model to learn how to align cross modal information is possible. Referring to the example in Fig.2, the video is paired with the sentence ”a boy is performing for an audience.”, where ”boy”, ”performing”, ”audience” are the keywords and can be categorized as ”entity”, ”action”, ”entity” accordingly. We design several experts to learn corresponding representations independently. In addition, a gating module is employed to measure their importance score and then strengthen the representation of the fusion expert. Such innovation brings little parameters and computations increment and surpassing the previous Stateof-The-Art (SOTA) method on various benchmarks. Previous work has taken a similar approach, either by simple part-of-speech tagging or by wielding multi-dimensional features on the video. HGR (Chen et al. 2020) and HCGC (Jin et al. 2021) hypothesize that a text can be constructed into a hierarchical semantic graph structure, where lie sentence, action, entity embedding in the top, second, third level node, respectively. T2VLAD (Wang, Zhu, and Yang 2021) extracts features from the aspects of scene and action, and performs similarity matching with the representations of each local token and the global sentence, while HiT (Liu et al. 2021) conducts cross-matching between featurelevel and semantic-level embedding. But they don’t simultaneously decompose the video and text to conduct deep alignment, from where we proposed the multi-stream multi-task (Ruder 2017) architecture, as shown in Fig.2 The heterogeneity of contents and dual optimal-match hypothesis. Another important contribution of this paper is the proposed problem that semantic and visual modals usually express in a different range of content. The example shown in Fig.1, denotes the comparisons of the process calculating the final probability matrix for Video-to-Text retrieval. Although each video describes specific and explicit content, the corresponding text can be unspecific and fuzzy, which harms model training. The original method conducts the softmax for every single retrieval, ignoring the potential cross-retrieval information and leading to a confusing result. To solve this, we propose the dual optimal-match hypothear X iv :2 10 9. 04 29 0v 2 [ cs .C V ] 1 3 Se p 20 21 A woman is decorating her finger nail A woman is mixing nail polish and putting an egg into it A girl is painting easter designs on nails