there is a wide range of applications in video retrieval and indexing grabbing researcher's attention. The following paper includes tutorial as well as a description of the background to common approaches in visual content-based video retrieval and indexing. The focal point is on the mechanisms of video shot boundary detection, key frame extraction, structure analysis, and scene segmentation, along with extraction of key frame features, static features, motion features, object features, video data mining, video annotation, video retrieval, including similarity measure, query interfaces, relevance feedback, and video browsing. Ultimately, directions for future work will be proposed.

With the development of multimedia data types and available bandwidth there is huge demand of video retrieval systems, as users shift from text based retrieval systems to content based retrieval systems. Selection of extracted features play an important role in content based video retrieval regardless of video attributes being under consideration. These features are intended for selecting, indexing and ranking according to their potential interest to the user. Good features selection also allows the time and space costs of the retrieval process to be reduced. This survey reviews the interesting features that can be extracted from video data for indexing and retrieval along with similarity measurement methods. We also identify present research issues in area of content based video retrieval systems.

With recent uptake in the usage of mobile devices, such as smartphones and tablets, increasing at an exponential rate, these devices have become part of everyday life. This high yield of information access comes at a cost. With still limited input metrics, it is prudent to develop content based techniques to filter the amount of content that is returned, for example, from search requests to video search engines. In addition, such handheld devices are used by a highly heterogeneous user community, including people with little or no experience. In this work, we focus on the latter, i.e. such casual users (‘novices’), and target video search and retrieval. We begin by examining new methods of developing related Content-Based Multimedia Information Retrieval systems for novices on handheld tablet devices. We analyze the shortcomings of traditional desktop systems which favor the expert user formulating complex queries and focus on the simplicity of design and interaction on tablet devices. We create and test three prototype demonstrators over three years of the TRECVid known item search task in order to determine the best features and appropriate usage to attain both high quality, usability, and precision from our novice users. In the first experiment, we determine that novice users perform similarly to an expert user group, one major premise of this research. In our second experiment, we analyze methods which can be applied automatically to aid novice users, thus enhancing their search performance. Our final experiment deals with different visualization approaches which can further aid the users. Overall, our results show that each year our systems made an incremental improvement. The 2011 TRECVid system performed best of all submissions in that year, despite the reduced complexity, enabling novice users to perform equally well as experts and experienced searchers.

While video content is often stored in rather large files or broadcasted in continuous streams, users are often interested in retrieving only a particular passage on a topic of interest to them. It is, therefore, necessary to split video documents or streams into shorter segments corresponding to appropriate retrieval units. We propose here a method for the automatic segmentation of TV news videos into stories. A-multiple-descriptor based segmentation approach is proposed. The selected multimodal features are complementary and give good insights about story boundaries. Once extracted, these features are expanded with a local temporal context and combined by an early fusion process. The story boundaries are then predicted using machine learning techniques. We investigate the system by experiments conducted using TRECVID 2003 data and protocol of the story boundary detection task, and we show that the proposed approach outperforms the state-of-the-art methods while requiring a very small amount of manual annotation.

We present in this article a video OCR system that detects and recognizes overlaid texts in video as well as its application to person identification in video documents. We proceed in several steps. First, text detection and temporal tracking are performed. After adaptation of images to a standard OCR system, a final post-processing combines multiple transcriptions of the same text box. The semi-supervised adaptation of this system to a particular video type (video broadcast from a French TV) is proposed and evaluated. The system is efficient as it runs 3 times faster than real time (including the OCR step) on a desktop Linux box. Both text detection and recognition are evaluated individually and through a person recognition task where it is shown that the combination of OCR and audio (speaker) information can greatly improve the performances of a state of the art audio based person identification system.

We present a video shot boundary detection (SBD) algorithm that spots discontinuities in visual stream by monitoring video frame trajectories on Self-Organizing Maps (SOMs). The SOM mapping compensates for the probability density differences in the feature space, and consequently distances between SOM coordinates are more informative than distances between plain feature vectors. The proposed method compares two sliding best-matching unit windows instead of just measuring distances between two trajectory points, which increases the robustness of the detector. This can be seen as a variant of the adaptive threshold SBD methods. Furthermore, the robustness is increased by using a committee machine of multiple SOM-based detectors. Experimental evaluation made by NIST in the TRECVID evaluation confirms that the SOM-based SBD method works comparatively well in news video segmentation, especially in gradual transition detection.

We build a fast video shot retrieval system in the context of the NIST TREC Video 2009 evaluation campaign. We compare our efficient Profile Entropy Features (PEF) to usual features, using various classifiers. These PEF are derived using the projection in the horizontal and vertical orientations. These features are then fed to SVM or KNNG classifiers to produce the keyframe ranks, from which we can get the shot ranks. The experimental results show that our PEF features outperform other features such as EDGE, GABOR, HSV, and so on. Moreover, PEF are very compact and fast to compute, and thus may be improved in further video retrieval systems.

Visual concept detection is one of the most active research areas in multimedia analysis. The goal of visual concept detection is to assign to each elementary temporal segment of a video, a confidence score for each target concept (e.g. forest, ocean, sky, etc.). The establishment of such associations between the video content and the concept labels is a key step toward semantics-based indexing, retrieval, and summarization of videos, as well as deeper analysis (e.g., video event detection). Due to its significance for the multimedia analysis community, concept detection is the topic of international benchmarking activities such as TRECVID. While video is typically a multi-modal signal composed of visual content, speech, audio, and possibly also subtitles, most research has so far focused on exploiting the visual modality. In this chapter, we introduce fusion and text analysis techniques for harnessing automatic speech recognition (ASR) transcripts or subtitles to improve the results of visual concept detection. Since the emphasis is on late fusion, the introduced algorithms for handling text and the fusion can be used in conjunction with standard algorithms for visual concept detection. We test our techniques on the TRECVID 2012 Semantic indexing (SIN) task dataset, which is made of more than 800 h of heterogeneous videos collected from Internet archives.

Video summarization refers to the process of recapitulating video stream by producing an abstract of the salient keyframes that could cover its overall content. However, an efficient video summarization requires an efficient video Shot Boundary Detection (SBD) and keyframes extraction. In this backdrop, this paper presents a novel and efficient approach for video SBD and keyframes extraction that will lead in the summarizing video. Meanly, Motion SIFT-Distribution Histogram (MoSIFT-DH) is extracted from the frames as a Glocal feature. The shot boundaries are detected using an adaptive threshold for the computed distance of MoSIFT-DH of the consecutive frames. Furthermore, keyframe representing the salient content of each segmented shot is extracted using entropy based singular values. Finally, the summarizing video is generated, by combining all the extracted keyframes. Our experiments on various videos indicate that our method can efficiently detect shot boundaries under different levels of illumination, camera operations and motion effects.

Video indexing and retrieval have a wide spectrum of promising applications, motivating the interest of researchers worldwide. This paper offers a tutorial and an overview of the landscape of general strategies in visual content-based video indexing and retrieval, focusing on methods for video structure analysis, including shot boundary detection, key frame extraction and scene segmentation, extraction of features including static key frame features, object features and motion features, video data mining, video annotation, video retrieval including query interfaces, similarity measure and relevance feedback, and video browsing. Finally, we analyze future research directions.

Users are often interested in retrieving only a particular passage on a topic of interest to them. It is therefore necessary to split videos into shorter segments corresponding to appropriate retrieval units. We propose here a method based on a local temporal context for the segmentation of TV news videos into stories. First, we extract multiple descriptors which are complementary and give good insights about story boundaries. Once extracted, these descriptors are expanded with a local temporal context and combined by an early fusion process. The story boundaries are then predicted using machine learning techniques. We investigate the system by experiments conducted using TRECVID 2003 data and protocol of the story boundary detection task and we show that the extension of multimodal descriptors by a local temporal context approach improves results and our method outperforms the state of the art.

Today, large volume of digital videos has become available online to the masses. There is the drastic growth in the amount of the multimedia data, because of the improvements in data storage, acquisition and communication technologies, which are supported by improvements in processing of audio and video signals. People are looking for the videos that contain certain information of their interest. Such a search is facilitated by Content Based Video Retrieval methods. Three major steps of CBVR are, dividing the videos into the segments, extracting the features from video, and after that retrieving videos based on information in query. Out of these 3 steps segmentation of video is most prominent step as the retrieval results are based on the segmentation boundaries. Segmentation divides the video into shots. A shot is a segment of the video that consists of one continuous action in time domain. The boundaries of such shots can be detected by using various different techniques like histogram, discrete cosine transforms, motion vector, edge tracking, and block matching methods. But due to the motion of objects and camera these shot boundaries can be detected falsely. This paper presents a comparative study of different methods/algorithms that have been proposed in literature.

This year we conducted experiments on shot boundary detection and rushes video summarisation. For the shot boundary determination task, we focused on detection of ‘cut’. The approach calculated the ‘exclusive or’ of two frames in the grey scale in order to measure the amount of discontinuity at a pixel level between two shots. Five runs were submitted with different sets of parameters, resulting in the performance of as high as 87% recall and 85% precision. For the rushes video task, the summary duration was x ed at 4% of the video length. We joined a number of continuous frames that were extracted from the middle of each shot detected. We submitted a single run, resulting in the average level performance.

This thesis presents a new system for video auto tagging which aims at correcting the tags provided by users for videos uploaded on the Internet. Most existing auto-tagging systems rely mainly on the textual information and learn a great number of classifiers (on per possible tag) to tag new videos. However, the existing user-provided video annotations are often incorrect and incomplete. Indeed, users uploading videos might often want to rapidly increase their video’s number-of-view by tagging them with popular tags which are irrelevant to the video. They can also forget an obvious tag which might greatly help an indexing process. In this thesis, we limit the use this questionable textual information and do not build a supervised model to perform the tag propagation. We propose to compare directly the visual content of the videos described by different sets of features such as SIFT-based Bag-Of-visual-Words or frequent patterns built from them. We then propose an original tag correction strategy based on the frequency of the tags in the visual neighborhood of the videos. We have also introduced a number of strategies and datasets to evaluate our system. The experiments show that our method can effectively improve the existing tags and that frequent patterns build from Bag-Of-visual-Words are useful to construct accurate visual features

This thesis describes research into speaker diarization for recorded meetings. It explores the algorithms and the implementation of an off-line speaker segmentation and clustering system for meetings that have been recorded using one microphone. Speaker diarization is defined as a process of partitioning a spoken record into speaker-homogeneous regions. The meeting record contains different kinds of noise and the length of the noise varies significantly. The average speech-turn is short and the number of speakers is unknown. To reduce the influence of these aural characteristics on the performance of the speaker diarization system, this thesis proposed four new algorithms. First, a new speech activity detection method, which adjusts the non-speech model complexity according to the noise length ratio. Second, a new speaker change point detection measure was derived based on the Fisher Linear Discriminate Analysis to help detect short speaker turns. Third, the Equal Weight Penalty Criterion was formulated as a new model complexity selection criterion to train both the speakers' models and the Universal Background Model (UBM). It contains two penalty terms, one penalizes the model dimensions and removes mixtures with small mixing probability, the other penalizes the Kullback Leibler divergence between the prior and posterior distribution of the mixing parameters, removing those components that share the same location. This criterion can be adjusted by the prior distribution parameter delta, which controls how many components are used in the model. Fourth, a weight and mean adaptation method was developed to adapt potential speaker models from the UBM. In addition, a potential speaker merging termination scheme, based on the Normalized Cuts, was introduced into the system. Combining all the new techniques derived in this thesis together, the error rate of the baseline system was reduced from 18.61% to 9.24% on the development set, 18.89% to 10.50% on the evaluation set from AMI corpus, and 21.35% to 15.48% on the evaluation set from ISL corpus. When using the Normalized Cuts based potential speaker merging termination scheme, the error rate of the baseline system was reduced 18.61% to 10.33% on the development set, 18.89% to 9.99% on the evaluation set from AMI corpus, and 21.35% to 13.70% percentage points on the evaluation set from ISL corpus.

This thesis combines audio-analysis with computer vision to approach Music Information Retrieval (MIR) tasks from a multi-modal perspective. This thesis focuses on the information provided by the visual layer of music videos and how it can be harnessed to augment and improve tasks of the MIR research domain. The main hypothesis of this work is based on the observation that certain expressive categories such as genre or theme can be recognized on the basis of the visual content alone, without the sound being heard. This leads to the hypothesis that there exists a visual language that is used to express mood or genre. In a further consequence it can be concluded that this visual information is music related and thus should be beneficial for the corresponding MIR tasks such as music genre classification or mood recognition. A series of comprehensive experiments and evaluations are conducted which are focused on the extraction of visual information and its application in different MIR tasks. A custom dataset is created, suitable to develop and test visual features which are able to represent music related information. Evaluations range from low-level visual features to high-level concepts retrieved by means of Deep Convolutional Neural Networks. Additionally, new visual features are introduced capturing rhythmic visual patterns. In all of these experiments the audio-based results serve as benchmark for the visual and audio-visual approaches. The experiments are conducted for three MIR tasks Artist Identification, Music Genre Classification and Cross-Genre Classification. Experiments show that an audio-visual approach harnessing high-level semantic information gained from visual concept detection, outperforms audio-only genre-classification accuracy by 16.43%.

This paper summarizes the collaboration of the LIA and CLIPS laboratories on speaker diarization of broadcast news during the spring NIST Rich Transcription 2003 evaluation campaign (NIST-RTO03S). The speaker diarization task consists of segmenting a conversation into homogeneous segments which are then grouped into speaker classes. Two approaches are described and compared for speaker diarization. The first one relies on a classical two-step speaker diarization strategy based on a detection of speaker turns followed by a clustering process, while the second one uses an integrated strategy where both segment boundaries and speaker tying of the segments are extracted simultaneously and challenged during the whole process. These two methods are used to investigate various strategies for the fusion of diarization results. Furthermore, segmentation into acoustic macro-classes is proposed and evaluated as a priori step to speaker diarization. The objective is to take advantage of the a priori acoustic information in the diariza-tion process. Along with enriching the resulting segmentation with information about speaker gender,

This paper presents the Shot Boundary Detection system developed by LaBRI in the context of “Rough Indexing” paradigm. We work on compressed streams and we use only I and P frames information, (DC coefficients of I-Frames, motion vectors of PFrames and DC coefficients of prediction error) which allow us to be faster than many equivalent systems (10 times faster than real-time on TRECVID2003 test set, and 3 times faster on 2004, because MPEG files structure is composed of only I and P frames). In this context the application was not developed to classify shot change transition effects, the initial goal was to allow a real-time and intelligent browsing in video content for common users. The detection is performed in two stages: - Robust Global Camera Motion Estimation - Detection of P-Frame peaks (computation of motion and frame statistics), and of I-Frames (measuring similarity on successive compensated I frames). As we work with two types of frames (I and P), we associate two statistical models which give us two sets of ratio and threshold to calibrate the detector. The first TRECVID participation of LaBRI implies an evolution of the application for transitions effects distinction, which induces two new thresholds to calibrate. We generally obtain equivalent values of Recall and Precision (0.72 on TRECVID 2003 test set). On TRECVID 2004 test set we obtain as best runs ri-3: 0.723(Recall) and 0.606(Precision); and ri-4: 0.703(Recall) and 0.635(Precision).

This paper discusses open problems of detecting shot boundaries for music videos. The number of shots per second and the type of transition are considered to be a discriminating feature for music videos and a potential multi-modal music feature. By providing an extensive list of effects and transition types that are rare in cinematic productions but common in music videos, we emphasize the artistic use of transitions in music videos. By the use of examples we discuss in detail the shortcomings of state-of-the-art approaches and provide suggestions to address these issues.

This paper describes the first steps of CLIPS/IMAG on the TREC video story segmentation task. We mostly describe the multi-modal features used and their respective performance for the story segmentation task. These features are based on the audio, video and text modalities. The preliminary system, which has the advantage to be relatively free with respect to the use of training data, is also presented in this paper. First experiments on the TRECVID 2003 evaluation set lead to a recall rate of 0.613 and a precision rate of 0.467. We plan to participate to the official TRECVID 2004 story segmentation task with this system