Multi-Modality Transfer Based on Multi-Graph Optimization for Domain Adaptive Video Concept Annotation

Multi-modality, the unique and important property of video data, is typically ignored in existing video adaptation processes. To solve this problem, we propose a novel approach, named multi-modality transfer based on multi- graph optimization (MMT-MGO) in this paper, which leverages multi-modality knowledge generalized by auxiliary classifiers in the source domain to assist multi-graph optimization (a graph-based semi-supervised learning method) in the target domain for video concept annotation. To our best knowledge, it is the first time to introduce multi-modality transfer into domain adaptive video concept detection and annotation. Moreover, we propose an efficient incremental extension scheme to sequentially estimate a small batch of new emerging data without modifying the structure of multi-graph scheme. The proposed scheme can achieve a comparable accuracy with that of the brand-new round optimization which combines these data with the data corpus for the nearest round optimization, while the time for estimation has been greatly reduced. Extensive experiments over TRECVID2005 and 2007 data sets demonstrate the effectiveness of both the multi-modality transfer scheme and the incremental extension scheme.