With the rising prevalence of social media, coupled with the ease of sharing images, people with specific needs and applications such as known item search, multimedia question answering, etc., have started searching for visual content, which is expressed in terms of complex queries. A complex query consists of multiple concepts and their attributes are arranged to convey semantics. It is less effective to answer such queries by simply appending the search results gathered from individual or subsets of concepts present in the query. In this paper, we propose to exploit the query constituents and relationships among them. The proposed approach finds image-query relevance by integrating three models - the linguistic pattern-based textual model, the visual model, and the cross modality model. We extract linguistic patterns from complex queries, gather their related crawled images, and assign relevance scores to images in the corpus. The relevance scores are then used to rank the images. We experiment on more than 140k images and compare the NDCG@n scores with the state-of-the-art image ranking methods for complex queries. Also, ranking of images obtained by our approach outperforms than that of obtained by a popular search engine.

With the rapid growth of multimedia application technologies and network technologies, especially the proliferation of Web 2.0 and digital cameras, there has been an explosion of images and videos in the Internet. For example, the volume of videos uploaded to the YouTube every minute is amounting to 48 hours by May 2011, having doubled in the last two years. Such huge video collections hold useful yet implicit and nontrivial knowledge about various domains. To manage and utilize these resources effectively, video concept detection becomes a very important subject of intensive research by a large research community (Over et al., 2008). It is an integral part of visual data mining that is automatically extracting such knowledge from the huge unstructured visual data. It aims to automatically annotate video shots or keyframes with respect to a semantic concept (Tang et al., 2012). Ranging from objects like airplane and car to scenes like urban street and sky, semantic concepts serve as good intermediate semantic features for video content indexing and understanding, and thus, spurring much research attention (Jiang et al., 2010; Naphade & Smith, 2004; Snoek et al., 2006; Zheng et al., 2008). Essentially, concept detection is a classification task, in which a binary classifier is usually learned to predict the presence of a certain concept in a video shot or keyframe (image). Traditional concept detection methods are mainly global classification: use supervised machine learning techniques, such as single Support Vector Machine (SVM), etc., over whole training dataset.

With the explosive growth of web videos on the Internet, it becomes challenging to efficiently browse hundreds or even thousands of videos. When searching an event query, users are often bewildered by the vast quantity of web videos returned by search engines. Exploring such results will be time consuming and it will also degrade user experience. In this paper, we present an approach for event driven web video summarization by tag localization and key-shot mining. We first localize the tags that are associated with each video into its shots. Then, we estimate the relevance of the shots with respect to the event query by matching the shot-level tags with the query. After that, we identify a set of key-shots from the shots that have high relevance scores by exploring the repeated occurrence characteristic of key sub-events. Following the scheme in [6] and [22], we provide two types of summaries, i.e., threaded video skimming and visual-textual storyboard. Experiments are conducted on a corpus that contains 60 queries and more than 10 000 web videos. The evaluation demonstrates the effectiveness of the proposed approach.

With the explosive growth of multimedia contents on the internet, multimedia search has become more and more important. However, users are often bewildered by the vast quantity of information content returned by the search engines. In this scenario, Multimedia Question-Answering (MMQA) emerges as a way to return precise answers by leveraging advanced media content and linguistic analysis as well as domain knowledge. This paper performs an initial study on exploring large scale data for MMQA. First, we construct a web video dataset and discuss its query strategy, statistics, feature description and groundtruth. We then conduct experiments based on the dataset to answer definition event questions using three schemes. We finally conclude the study with discussion for future work.

With the advent of video sharing websites, the amount of videos on the internet grows rapidly. Web video categorization is an efficient methodology for organizing the huge amount of videos. In this paper we investigate the characteristics of web videos, and make two contributions for the large scale incremental web video categorization. First, we develop an effective semantic feature space Concept Collection for Web Video with Categorization Distinguishability (CCWV-CD), which is consisted of concepts with small semantic gap, and the concept correlations are diffused by a novel Wikipedia Propagation (WP) method. Second, we propose an incremental support vector machine with fixed number of support vectors (n-ISVM) for large scale incremental learning. To evaluate the performance of CCWV-CD, WP and n-ISVM, we conduct extensive experiments on the dataset of 80,021 most representative videos on a video sharing website. The experiment results show that the CCWV-CD and WP is more representative for web videos, and the n-ISVM algorithm greatly improves the efficiency in the situation of incremental learning.

We submitted six interactive search runs for TRECVID2007, including 2 single user and 4 collaborative runs. In one single-user run (FXPAL MMA) the searchers had access to all resources including text search, text similarity, image similarity, and concept similarity search. In the other single-user submission (FXPAL MMV) the transcripts were not available and only features derived directly from the audio-visual content were available. This is a typical baseline for the manual and automatic search systems, but has been a less popular variation among the interactive search submissions. The other submitted runs (FXPAL CO*) employed a novel real-time, multi-user, collaborative search system. The complete set of submitted runs in priority order with system names and brief descriptions:

We propose a novel scheme to address video concept learning by leveraging social media, one that includes the selection of web training data and the transfer of subspace learning within a unified framework. Due to the existence of cross-domain incoherence resulting from the mismatch of data distributions, how to select sufficient positive training samples from scattered and diffused social media resources is a challenging problem in the training of effective concept detectors. In this paper, given a concept, the coherent positive samples from web images for further concept learning are selected based on the degree of image coherence. Then, by exploiting both the selected dataset and video keyframes, we train a robust concept classifier by means of a transfer subspace learning method. Experiment results demonstrate that the proposed approach can achieve constant overall improvement despite cross-domain incoherence.

We describe our experiments for the High-level Feature Extraction (FE) and Search (SE) tasks. We submitted two automatic runs to the FE task, the first one (MMIS alexei) was based on a probabilistic approach while the second (MMIS ainhoa) was an enhanced version that used background knowledge in the form of statistical co-occurrence of annotation keywords. While previous applications of this approach to other datasets have performed quite well, our results in TRECVID 2008 are not so good. In particular, the performance of the second run was limited by the small vocabulary. For the SE task we submitted two runs: a similarity-based media search (MMIS media) and the required text-only search (MMIS text). The similarity search, using media content, had better precision than the text-only search but had difficulties with some types of queries (e.g., motion-based). Overall, participation in the TRECVID evaluation was a valuable learning experience for our group.

To exploit the hidden group structures of data and thus detect concepts in videos, this paper proposes a novel group sparse ensemble learning approach based on Automatic Group Sparse Coding (AutoGSC). We first adopt AutoGSC to learn both a common dictionary over different data groups and an individual group-specific dictionary for each data group which can help us to capture the discrimination information contained in different data groups. Next, we represent each data instance by using a sparse linear combination of both dictionaries. Finally, we propose an algorithm to use the reconstruction errors of data instances to calculate the ensemble gating function for ensemble construction and fusion. Experiments on the TRECVid 2008 benchmark show that the ensemble learning proposal achieves promising results and outperforms existing approaches.

This work presents a novel sparse ensemble learning scheme for concept detection in videos. The proposed ensemble first exploits a sparse non-negative matrix factorization (NMF) process to represent data instances in parts and partition the data space into localities, and then coordinates the individual classifiers in each locality for final classification. In the sparse NMF, data exemplars are projected to a set of locality bases, in which the non-negative superposition of basis images reconstructs the original exemplars. This additive combination ensures that each locality captures the characteristics of data exemplars in part, thus enabling the local classifiers to hold reasonable diversity in their own regions of expertise. More importantly, the sparse NMF ensures that an exemplar is projected to only a few bases (localities) with non-zero coefficients. The resultant ensemble model is, therefore, sparse, in the way that only a small number of efficient classifiers in the ensemble will fire on a testing sample. Extensive tests on the TRECVid 08 and 09 datasets show that the proposed ensemble learning achieves promising results and outperforms existing approaches. The proposed scheme is feature-independent, and can be applied in many other large scale pattern recognition problems besides visual concept detection.

This paper reports our experiments for TRECVID 2008 tasks: high level feature extraction, search and contentbased copy detection. For the high level feature extraction task, we use the baseline features such as color moments, edge orientation histogram and local binary patterns with SVM classifiers and nearest neighbor classifiers. For the search task, we use different approaches including search by the baseline features and search by concept suggestion. And for the video copy detection task, we study two approaches that are based on the pattern of motions in feature point trajectories and matches of all frame pairs using normalized cross correlation. Our approaches can be considered as one of the baseline approaches for evaluation of these tasks. I. HIGH LEVEL FEATURE EXTRACTION

This paper presents our approaches and results of the four TRECVID 2008 tasks we participated in: high-level feature extraction, automatic video search, video copy detection, and rushes summarization. In high-level feature extraction, we jointly submitted our results with Columbia University. The four runs submitted through CityU aim to explore context-based concept fusion by modeling inter-concept relationship. The relationship is modeled not based on semantic reasoning, but by observing how concepts correlate to each other, either directly or indirectly, in LSCOM common annotation [1]. An observability space (OS) [2] is thus built on top of LSCOM [1] and VIREO374 [3] for performing concept fusion. Since 19 of the 20 concepts evaluated this year appeared in VIREO-374, we apply OS to re-rank the results of both old models from VIREO-374 and new models from a joint baseline submission with Columbia. A CityU-HK1: re-rank A CU-run5 using OS – both positive and negative correlated concepts are used. A CityU-HK2: re-rank A CU-run5 using OS – only positive correlated concepts are used. A CityU-HK3: re-rank old models from VIREO-374 using OS – both positive and negative correlated concepts are used. A CityU-HK4: re-rank old models from VIREO-374 using OS – only positive correlated concepts are used. In automatic search, we focus on concept-based video search. The search is beyond semantic reasoning, where we consider the fusion of detectors using concept semantics, co-occurrence, diversity, and detector robustness. Two runs are submitted based on the works in [2] and [4] respectively. F A 2 CityUHK1 1: multi-modality fusion of concept-based search (Run-2), query example based search (Run-4 and Run-5), and text baseline (Run-6). F A 2 CityUHK2 2: concept-based search by fusing semantics, observability, reliability and diversity of concept detectors [2]. F A 2 CityUHK3 3: concept-based search using semantics reasoning [4, 5]. F A 2 CityUHK4 4: query-by-example – using VIREO-374 detection scores as features. F A 2 CityUHK5 5: query-by-example – using motion histograms as features. F A 1 CityUHK6 6: text baseline. In content-based video copy detection, we adopt a recently proposed near-duplicate video detection method [6, 7] based on the matching of local keypoint features. We submitted three runs: CityUHK loose: we use cosine similarity of visual word histograms to generate candidate near-duplicate keyframe set. The set is further filtered by a recently proposed method called SR-PE [6]. CityUHK vkisect: same with CityUHK loose except that we use histogram intersection instead of cosine similarity for candidate keyframe set generation. CityUHK tight: similar to CityUHK loose, but we add in few more heuristical constraints. In BBC rushes summarization, we submitted one run using the same method with our last year’s submission [8]. 1 High-Level Feature Extraction (HLFE) This year, we jointly submitted our HLFE results with Columbia University. Detailed descriptions of the joint submissions can be found in the notebook paper of Columbia [9]. For the four runs submitted by CityU, we aim to test context-based concept fusion based on a linear space (observability space) built from the observation derived from manual concept annotation. 1.1 Concept Fusion with Observability Space The observability space (OS) is proposed to effectively model the co-occurrence relationship among concepts [2]. We refine the individual concept detectors by using simple and efficient linear weighted fusion of the target concepts with several peripherally related concepts, where both concept selection and fusion weights are determined by the OS. Given a concept set V of n concepts, we first construct a n×n concept observability matrix R where each entry rij represents the co-occurrence relationship of a concept pair (Ci, Cj), measured by Pearson product-moment (PM) correlation: rij = PM(Ci, Cj) = ∑|T | k=1(Oik − μi)(Ojk − μj) (|T | − 1)σiσj (1) where Oik is the observability of concept Ci in shot k, and μi and σi are the sample mean and standard deviation, respectively, of observing Ci in a training set T . We set Oik to 1 if Ci presents in shot k, and 0 otherwise. With R, basis vectors C of OS can be estimated by solving following equation

This paper focuses on the comparison between two fusion methods, namely early fusion and late fusion. The former fusion is carried out at kernel level, also known as multiple kernel learning, and in the latter, the modalities are fused through logistic regression at classifier score level. Two kinds of multilayer fusion structures, differing in the quantities of feature/kernel groups in a lower fusion layer, are constructed for early and late fusion systems, respectively. The goal of these fusion methods is to put each of various features into effect and mine redundant information of the combination of them, and then to develop a generic and robust semantic indexing system to bridge semantic gap between human concepts and these low-level visual features. Performance evaluated on both TRECVID2009 and TRECVID2010 datasets demonstrates that the systems with our proposed multilayer fusion methods at kernel level perform more stably to reach the goal than the classification-score-level fusion; the most effective and robust one with highest MAP score is constructed by early fusion with two-layer equally weighted composite kernel learning.

This paper describes our system for auto search and inter- active search in the known-item search (KIS) task in TRECVID 2010. KIS task aims to find an unique video answer for each text query. The shift from traditional video search has prompted a series of challenges in processing and searching techniques that developed over the past few years. For the automatic search task, our VisionGo system performs query expansion and analysis, then employs multi-modality features in- cluding metadata, automatic speech recognition (ASR) and high level feature (HLF) to retrieve a ranked list of results deemed most relevant to the text-only query. To further improve the search performance, we crawl an extension set of tags from Youtube to supplement to TRECVID metadata. For interactive search task, we propose a new feedback scheme based on both related samples and exclusive negative samples to boost the search performance. To accomplish this, we introduce three enhance- ments to our VisioGo system: a) related sample feedback algorithm that allows users to indicate related (but not relevant) shots to the query; b) exclusive negative sample selection approach; and c) clustered shot-icons for efficiently representing the whole content of the video. Results from TRECVID 2010 video test set indicate that the enhancements are effec- tive.

This paper describes our experiments for the high level feature extraction task in TRECVid 2008. We submitted the following six runs: • A jrs1 1: Baseline, early fusion, tuned SVMs. • A jrs2 2: Early fusion, SVMs without parameter tuning. • A jrs3 3: Late fusion, tuned SVMs. • A jrs4 4: Early fusion, transductive SVM (TSVM) learning using UniverSVM with unlabeled samples on the scale of 20% of test set size. • A jrs5 5: Early fusion, transductive SVM learning using SVM TSVM with unlabeled samples on the scale of the training set size. • A jrs6 6: Early fusion, transductive SVM learning using SVM TSVM with unlabeled samples on the scale of 20% of test set size. The experiments were designed to study both the performance of various content-based features in connection with classic early and late feature fusion as well as the influence of SVM parameter tuning and learning from unlabeled test data with different implementations of transductive support vector machines (TSVMs). The results show that the time-consuming parameter tuning improves precision only marginally. Compared to standard SVMs, TSVMs did not provide overall improvement and only slight benefits for concepts with a small number of training samples.

The overwhelming amount of multimedia entities shared over the web has given rise to the need for semantic identification and classification of these entities. Numerous research efforts have tackled this problem by developing advanced content analysis techniques as well as leveraging on readily available tags, scripts, and blogs related to these multimedia entities. However, in many cases, especially for event detection and action recognition, the research efforts were hampered by the lack of large scale publicly available benchmarks. To address this problem, this paper presents a large-scale movie corpus named MovieBase that covers full length feature movies as well as huge volume of movie-related video clips downloaded from YouTube. The corpus is designed for research in event detection and action recognition. It offers over 71 hours of videos with a total of 69,129 shots. The corpus has been hand-labeled according to 7 audio and 11 visual concept tags to semantically define 11 event categories under the romantic and violence scenes. The corpus comes with a set of pre-extracted low-level visual, motion, audio as well as high-level features. Related results are furnished as a baseline for the movie event detection task.

The bag-of-words (BoW) has been widely regarded as the most successful algorithms for content-based image related tasks, such as large scale image retrieval, classification, and object categorization. Large visual words acquired by BoW quantization through large vocabulary or codebooks have been receiving much attention in the past years. However, not only construction of large vocabulary but also the quantization process impose a heavy burden in terms of time and memory complexities. In order to tackle this issue, we propose an efficient hierarchical BoW (HBoW) to achieve large visual words through quantization by a compact vocabulary instead of large vocabulary. Our vocabulary is very compact since it is only composed of two small dictionaries which is learned through segmental sparse decomposition of local features. To generate the BoW with large size, we first divide the local features into two half parts, and use the two small dictionaries to compute their sparse codes. Then, we map the two indices of the maximum elements of the two sparse codes to a large set of visual words based upon the fact that data with similar properties will share the same base weighted with the largest sparse coefficient. To further make similar patches have higher probability of select the same dictionary base to get similar BoW vectors, we propose a novel collaborative dictionary learning method by imposing the similarity regularization factor together with the row sparsity regularization across data instances during group sparse coding. Additionally, based on index combination of top-2 large sparse codes of local descriptors, we propose a soft BoW assignment method so that our proposed HBoW can tolerate different word selection for similar patches. By employing the inverted file structure built through our HBoW, K-nearest neighbors (KNN) can be efficiently retrieved. After incorporation of our fast KNN search into the SVM-KNN classification method, our HBoW can be used for efficient image classification and logo recognition. Experiments on serval well-known datasets show that our approach is effective for large scale image classification and retrieval.

Tags associated with web videos play a crucial role in organizing and accessing large-scale video collections. However, the raw tag list (RawL) is usually incomplete, imprecise and unranked, which reduces the usability of tags. Meanwhile, compared with studies on improving the quality of web image tags, tags associated with web videos are not studied to the same extent. In this paper, we propose a novel web video tag enhancement approach called video retagging, which aims at producing the more complete, precise, and ranked retagged tag list (RetL) for web videos. Given a web video, video retagging first collect its textually and visually related neighbor videos. All tags attached to the neighbors are treated as possible relevant ones and then RetL is generated by inferring the degree of relevance of the tags from both global and video-specific perspectives, using two different graph based models. Two kinds of experiments, i.e., application-oriented video search and categorization and user-based subjective studies are carried out on a large-scale web video dataset, which demonstrate that in most cases, RetL is better than RawL in terms of completeness, precision and ranking.

Surveillance Event Detection Semantic event detection in the huge amount of surveillance video in both retrospective and real-time styles is essential to a variety of higher-level applications in the public security. In TRECVID 2010, to overcome the limitations of the traditional human action analysis method with human detection/tracking and domain knowledge, we evaluate the general framework for multiple human behaviors modeling with the philosophy of bag of spatiotemporal feature (BoSTF). The brief

Shot boundary detection The shot boundary detection system in 2007 is basically the same as that of last year. We make three major modifications in the system of this year. First, CUT detector and GT detector use block based RGB color histogram with the different parameters instead of the same ones. Secondly, we add a motion detection module to the GT detector to remove the false alarms caused by camera motion or large object movements. Finally, we add a post-processing module based on SIFT feature after both CUT and GT detector. The evaluation results show that all these modifications bring performance improvements to the system. The brief introduction to each run is shown in the following table: Run_id Description Thu01 Baseline system: RGB4_48 for CUT and GT detector, no motion detector, no sift post-processing, only using development set of 2005 as training set Thu02 Same algorithm as thu01, but with RGB16_48 for CUT detector, RGB4_48 for GT detector Thu03 Same algorithm as thu02, but with SIFT post-processing for CUT Thu04 Same algorithm as thu03, but with Motion detector for GT Thu05 Same algorithm as thu04, but with SIFT post-processing for GT Thu06 Same algorithm as thu05, but no SIFT processing for CUT Thu09 Same algorithm as thu05, but with different parameters thu11 Same algorithm as thu05, but with different parameters Thu13 Same algorithm as thu05, but with different parameters Thu14 Same algorithm and parameters as thu05, but trained with all the development data from 2003-2006 High-level feature extraction We try a novel approach, Multi-Label Multi-Feature learning (MLMF learning) to learn a joint-concept distribution on the regional level as an intermediate representation. Besides, we improve our Video diver indexing system by designing new features, comparing learning algorithms and exploring novel fusion algorithms. Based on these efforts in improving feature, learning and fusion algorithms, we achieve top results in HFE this year.