Deals with the representation, storage and retrieval of unstructured data Topics of interest: systems, languages, retrieval, user interfaces, data visualization, distributed data sets Classical IR deals mainly with text The evolution of multimedia databases and of the web have given new interest to IR

vii CHAPTER

that have been already implemented in cultural heritage or we think are useful for the GRAVITATE project. These fields are relatively disjoint, and the research and development challenge of GRAVITATE is precisely to merge them. After the review of the current literature on these fields, we end the report with common remarks on possible or plausible crossconnections that suggest themselves. These considerations will be refined for the Roadmap for Research deliverable. D3.1

of a dissertation at the University of Miami. Dissertation supervised by Professor Mei-Ling Shyu. No. of pages in text. (153) The development in information science has enabled an explosive growth of data, which attracts more and more researchers to engage in the field of big data analytics. Noticeably, in many real-world applications, large amounts of data are imbalanced data since the events of interests occur infrequently. Classification of imbalanced data is an important research problem as lots of real-world datasets have skewed class distributions in which the majority of instances (examples) belong to one class and far fewer instances belong to the others. A classifier induced from an imbalanced dataset is more likely to be biased towards the majority classes and shows very poor classification accuracy on the minority classes. While in many applications, the minority instances actually represent the concept of interest (e.g., fraud in banking operations, abnormal cell in medical data, etc.), and the detection of these rare events has become more important. Despite extensive research efforts, rare event mining remains one of the most challenging problems in information retrieval, especially for multimedia big data. To tackle this challenge, in this dissertation, we propose an extended deep learning approach to achieve promising performance in classifying largely skewed multimedia dataset. Specifically, we investigate the integration of bootstrapping methods and a state-of-the-art deep learning approach, Convolutional Neural Networks (CNNs), with extensive empirical studies. Considering the fact that deep learning approaches such as CNNs are usually computationally expensive, we propose to feed low-level features to CNNs and prove its feasibility in achieving promising performance while saving a lot of training time. Furthermore, since big training datasets are required to train CNNs, we propose to extract features from pre-trained CNN models and feed those features to another full connected neural network. Implementations in big data environments show promising performance of our model in handling big datasets with respect to feasibility and scalability. In order to further improve the classification results and bridge the semantic gap between high-level concepts and low-level visual features, correlation discovery in semantic concept mining is worth exploring. Though inter-concept correlations have been utilized to address this issue recently, the very small number of instances in the minority classes often lead to the detection of imprecise correlations and unsatisfactory classification results. Meanwhile, correlation discovery is a computationally intensive task in the sense that it requires a deep analysis of very large and growing repositories. This dissertation further proposes a novel concept correlation analysis strategy framework that utilizes the correlations between the retrieval scores and labels. By integrating the correlation information, the proposed framework can help imbalanced data classification and enhance rare class (event or concept) mining even with trivial scores from the minority classes. Not only deep learning but also numerous other classification algorithms have been developed for a variety of data types. However, it is nearly impossible for one classifier to perform the best in all kinds of datasets all the time. Therefore, ensemble learning models which aim to take advantages of different classifiers have received a lot of attentions recently. In this dissertation, a scalable classifier ensemble framework assisted by a set of “judgers” is also proposed to integrate the outputs from multiple classifiers for multimedia big data classification. Specifically, based on the confusion matrices of different classifiers, a set of judgers are organized into a hierarchically structured decision model. A testing instance is first input to different classifiers, and then the classification results are passed to the proposed hierarchical structured decision model to derive the final result. The ensemble system can be run on Spark, which is designed for big data processing. All the proposed components are evaluated on multimedia datasets containing different kinds of data. The experimental results show the effectiveness of our framework in classifying severely imbalanced data with promising performance, and demonstrate that the proposed classifier ensemble framework outperforms several state-of-the-art model fusion approaches. Furthermore, the proposed framework is applied to two realworld applications, i.e., deep learning based text data analysis on an Amazon review dataset and efficient large-scale stance Analysis in Twitter, and achieves promising results in both. In additional, we also design a web-based information retrieval system and identify several future directions that could be explored to further improve the current work. To my dear parents and grandparents

of a dissertation at the University of Miami. Dissertation supervised by Professor Mei-Ling Shyu No. of pages in text. (170) The prevalence of digital recording devices, the cheap cost of data storage as well as the convenience provided by the widely accessible Internet have created the demand to retrieve information according to users’ requests from multimedia data sources. However, the multimedia information retrieval task has several challenges that need to be addressed, such as bridging the semantic gap, modeling from imbalanced data sets, and utilizing inter-concept relationships to enhance the retrieval performance of an individual concept. To respond to the challenge of bridging the semantic gap, subspace modeling methods are proposed to address this issue as a classification task. The proposed subspace modeling methods construct a principal component (PC) subspace for each class, where the PCs are derived from the instances belonging to that class. The PCs are selected and ranked based on Fisher’s criterion to reduce the searching effort and an iterative searching is utilized to determine the best PC set. Subspace modeling methods are proposed in this dissertation, including multi-class subspace modeling (MSM), binary-class subspace modeling (BSM), and subspace modeling on global and local structures (SMGL). Comparative experiments show that MSM, BSM, and SMGL can outperform some other well-known algorithms on a number of benchmark data sets. To address the data imbalance challenge, two clustering-based subspace modeling methods called clustering-based subspace modeling (CLU-SUMO) and class selection and clusteringbased subspace modeling (CSC-SUMO) are proposed. K-means clustering and/or semantic concept labels are used to partition the majority class (usually the negative class) into several groups, each of which is merged with the minority class (usually the positive class) to form a much more balanced subset of the original data set. Then, the subspace model learned from the original data set is integrated with all the subspace models learned from the balanced subsets to form a classification framework. The experimental results on news and broadcast video data sets support the claim that the proposed CLU-SUMO and CSC-SUMO render better classification performance than some existing techniques that are commonly used to handle the data imbalance

ion In this paper, we describe our first participation for the semantic indexing task at TRECVID 2010 [1]. We focus on extraction multiple low-level feature sets and a fusion method. In our system, six features are extracted for all the predefined concepts from the keyframes, including global features (RGB color histogram, HSV color histogram, edge histogram, Grey Level Co-occurrence Matrix, GIST) and a local feature (gray-scale SIFT). SVM-based classifiers are trained by utilizing these features and multiple feature weighted fusion of the classification results are used as a baseline. In this year, only one run was submitted to “full” submission: F_A_IIPLA_Ritsu_CBVR_1: Multiple feature weighted fusion of classification results based on global features and local features are utilized. SVM classifiers are trained on the images provided by the collaborative annotation in TRECVID 2010.

concept of her. Collins and Jin [109] show that a ‘grandmother cell’ type representation could be information theoretically efficient provided it is accompanied by distributed coding type cells. Maass [110] shows that WTA is quite powerful compared to threshold and sigmoidal gates often used in traditional neural networks. It is shown that any Boolean function can be computed using a single k-WTA unit [110]. This is very interesting, as at least two-layered perceptron circuits are needed to compute complicated functions. They also showed that any continuous function can be approximated by a single soft WTA unit (A soft winner take all operation assumes values depending on the rank of corresponding input in linear order). Another advantage is the fact that approximate WTA computation can be done very fast (linear-size) in analog VLSI chips [111]. Thus, complex feed-forward multi-layered perceptron circuits can be replaced by a single competitive WTA stage leading to low power analog VLSI chips [110]. There have been many implementations of winner take all (WTA) computations in recurrent networks in the literature [112, 113]. Also there have been many analog VLSI implementations of these circuit [113, 114]. The WTA model implemented here is influenced by the WTA implementation on recurrent networks by Oster and Liu [113]. In their implementation, the neuron that receives spikes with the shortest inter-spike interval is the winner. But it is not 49 clear in their implementation how (starting from random weights) a new neuron can learn a new category. A modified version of winner take all (WTA) with Hebbian learning is implemented here to demonstrate how different neurons can learn different categories. WTA is applied on both the learning layers while training and is switched off while testing. The WTA is implemented as follows: 1) At every time step, find the post neuron with the least spike time difference tpost1tpost2. Note that these last two post-synaptic neuron spike times are easily available. This neuron is declared as the winner. 2) The winner inhibits the other neurons from firing by sending an inhibitory pulse to others. If the winner neuron has not learned any feature, it learns the new feature by the above Hebbian learning method. The neuron remains the winner unless another neuron has a lower spike time interval or a new image is presented. The above learning approach is followed except in the last (uppermost) learning layer where the winner is declared according to the supplied category/label information about the input image rather than the spike time difference. Thus the overall approach is like a semi-supervised approach, with unsupervised learning in the lower learning layer and supervised learning in the last learning layer. We are assuming that all membrane potentials are discharged at the onset of a stimulus. This can be achieved, for example, by a decay in the membrane potential. There have been several implementations of sparse coding schemes too. Földiák [105] shows how a layer of neurons can learn to sparse code by using Hebbian excitatory connections between the input and output units and anti-Hebbian learning among the output connections. Olshausen and Field [115] showed that minimizing an objective function of high sparseness and low reconstruction error on a set of natural images yields a set of basis functions similar to the Gabor like receptive fields of simple cells in primary visual cortex. One interesting study is by 50 Einhauser et al. [116], who developed a neural network model which could develop receptive field properties similar to the simple and complex cells found in visual cortex. The network could learn from natural stimuli obtained by mounting a camera to a cat’s head to approximate input to a cat’s visual system. Though, they did not use spiking neural network. 4.3 Neuronal and Synaptic Genesis Neurogenesis and synaptogenesis (birth of neurons or synapses) has been shown to occur in the brain in certain settings [117, 118]. It is also known that it does affect some learning and memory tasks [119-121]. Many studies have indicated that the brain is much more plastic than previously thought [122]. For example, brain scans of people who loose their limbs in accidents show that parts of the brain maps corresponding to the lost limb are taken over by the surrounding brain maps. This plasticity and change can be made possible by birth and death of neurons and connections. For efficient simulations it will be important to be able to add neurons and synapses, and to also remove them when necessary. We found that our network will benefit from modeling these processes. For illustration, we trained a network with the network architecture shown in the right plot of Figure 3-9 using our learning approach. The goal was to recognize handwritten digits from 0 to 9. More details about the network structure and training problem can be found in Section 6.4.1 Our focus here is on the first learning layer in the architecture which was trained in an unsupervised way. Initially the synaptic weights were set to random values in this layer. Figure 4-2 shows 2D-arrays of synapse values after training for this learning layer plotted as 8x8 projections onto the previous layer. The synaptic weights are plotted as intensities with white representing the highest synaptic strength and black the lowest. These arrays represent features that were learned. Note that some of them remained unchanged (remain random starting values) during training and could have as well been 51 removed from the simulation for efficiency. Or, better yet, we could have started with fewer synapses and added them as needed. This would ensure that unnecessary connections are not there and will save computer memory. Figure 4-2: Images of arrays of final synapse weights plotted as 8x8 projections to previous layer in a layer of network after learning. White represents the highest synaptic strength and black represents the lowest.

[This paper has been withdrawn by the publisher]. A novel method to identify the violent videos only with audio features is introduced. Most previous content-based image or video classification schemes apply the bag of words (BOW) or bag of visual words (BOVW), which employ multiple visual features to characterize image or video content. In our method, the bag of audio words (BOAW) is suggested to be built by effective audio features. Two reasons are considered here. First, audio features should have very special significance for violent videos. Second, the computational complexity of dealing with audio features is much lower than that of visual features. The MPEG-7 low level features such as Audio Spectrum-Centroid and Audio Spectrum-Spread, and the high level feature such as Audio Signature, are combined into one 44-dimensions vector in the BOAW model. The audio words are built from the vector by the clustering strategy, and support vector machine (SVM) with revised soft-weighting scheme is used to group the audio words features into two classes, i.e. the violent and non-violent. Experiments demonstrate that the proposed method can achieve good recall accuracy and precision accuracy on detecting violent videos. The method also can be applied to classify other types of videos.

[ANGLES] This project explores the potential of LIRE, an en existing Content-Based Image Retrieval (CBIR) system, when used to retrieve medical videos. These videos are recording of the live streams used by surgeons during the endoscopic procedures, captured from inside of the subject. The growth of such video content stored in servers requires search engines capable to assist surgeons in their management and retrieval. In our tool, queries are formulated by visual examples and those allow surgeons to re-find shots taken during the procedure. This thesis presents an extension and adaptation of Lire for video retrieval based on visual features and late fusion. The results are assessed from two perspectives: a quantitative and qualitative one. While the quantitative one follows the standard practices and metrics for video retrieval, the qualitative assessment has been based on an empirical social study using a semi-interactive web-interface. In particular, a thinking aloud test was applied to analyze if the user expectations and requirements were fulfilled. Due to the scarcity of surgeons available for the qualitative tests, a second domain was also addressed: videos captured at musical performances. These type of videos has also experienced an exponential growth with the advent of affordable multimedia smart phones, available to a large audience. Analogously to the endoscopic videos, searching in a large data set of such videos is a challenging topic.

Zusammenfassung Der Umfang, in dem multimediale Daten verfügbar sind, nimmt ständig zu. Dies gilt für das Internet ebenso wie für die Daten auf dem heimischen PC oder auch auf einem PDA. Eine wesentliche Herausforderung stellt dabei die Suche nach Medienobjekten dar. Der vorliegende Artikel geht auf grundlegende Konzepte und Problemstellungen zur Suche nach Medienobjekten und multimedialen Dokumenten ein und gibt einen Überblick über die für verschiedene Medientypen existierenden Ansätze. Abstract The extent to which multimedia data are available is steadily increasing. This is true for the Internet as well as for the data on a local PC or on a PDA. A major challenge in this respect is the search for media objects. The present article deals with fundamental concepts and issues of search for media objects and multimedia documents, and gives an overview of the approaches existing for the different media types.

Zusammenfassung 1 1 Overview 5 1.

Zusammenfassung 1

YAXIAO SONG: Multi-modal Surrogates for Retrieving and Making Sense of Videos: Is Synchronization between the Multiple Modalities Optimal? (Under the direction of Dr. Gary Marchionini) Video surrogates can help people quickly make sense of the content of a video before downloading or seeking more detailed information. Visual and audio features of a video are primary information carriers and might become important components of video retrieval and video sense-making. In the past decades, most research and development efforts on video surrogates have focused on visual features of the video, and comparatively little work has been done on audio surrogates and examining their pros and cons in aiding users’ retrieval and sense-making of digital videos. Even less work has been done on multi-modal surrogates, where more than one modality are employed for consuming the surrogates, for example, the audio and visual modalities. This research examined the effectiveness of a number of multi-modal surrogates, and investigated whether synchronization between the audio and visual channels is optimal. A user study was conducted to evaluate six different surrogates on a set of six recognition and inference tasks to answer two main research questions: (1) How do automatically-generated multi-modal surrogates compare to manuallygenerated ones in video retrieval and video sense-making? and (2) Does synchronization between multiple surrogate channels enhance or inhibit video retrieval and

Within our approach to big data, we reduce the number of images in video footage by applying a shot detection with a keyframe extraction of single frames. This can be followed by duplicate removal and face detection processes yielding to a further data reduction. Nevertheless, additional reductions steps are necessary in order to make the data manageable (searchable) for the end user in a meaningful way. Therefore, we investigated human inspired forgetting as a data reduction tool. We conducted an exploratory study on a subset of the remaining face data to examine patterns in the selection process of faces that are considered most memorable showing a potential of roughly above 75 % for elimination. The results of the study considered the quality and the size of the faces as important measures. In these terms, we finally show a connection to characteristics of state-of-the-art face detectors.

With user-generated video (UGV) becoming so popular on the Web, the availability of a reliable quality assessment (QA) measure of UGV is necessary for improving the users’ quality of experience in video-based application. In this paper, we explore QA of UGV based on how much irregular camera motion it contains with low-cost manner. A block-match based optical flow approach has been employed to extract camera motion features in UGV, based on which, irregular camera motion is calculated and automatic QA scores are given. Using a set of UGV clips from benchmarking datasets as a showcase, we observe that QA scores from the proposed automatic method and subjective method fit well. Further, the automatic method reports much better performance than the random run. These confirm the satisfaction of the automatic QA scores indicating the quality of the UGV when only considering visual camera motion. Furthermore, it also shows that the UGV quality can be assessed automatically for improving the end users quality of experience in video-based applications.

With the wide applications of vision based intelligent systems, image and video analysis technologies have attracted the attention of researchers in the computer vision field. In image and video analysis, human activity recognition is an important research direction. By interpreting and understanding human activities, we can recognize and predict the occurrence of crimes and help the police or other agencies react immediately. In the past, a large number of papers have been published on human activity recognition in video and image sequences. In this paper, we provide a comprehensive survey of the recent development of the techniques, including methods, systems, and quantitative evaluation of the performance of human activity recognition.

With the wide applications of video cameras in surveillance, video analysis technologies have attracted the attention from the researchers in computer vision field. In video analysis, human behavior recognition and understanding is an important research direction. By recognition and understanding the human behaviors, we can predict and recognize the happening of crimes and help to the police or other agencies to react immediately. In the past, large amount of intensive papers have been published on human behavior understanding in videos. Generally speaking, the procedure of human behavior understanding can be divided into the following stages: human segmentation and tracking, and human behavior recognition. In this paper, we provide a comprehensive survey of the recent development of all these stages. We will also discuss the difficulties in behavior understanding and identify possible future directions.

With the recent advancement in multimedia technologies, in conjunction with the rapid increase of the volume of digital video data and the growth of internet ; it has becom mandatory to have the hability browse and search through information stored in large multimedia databases. For this purpose, content based video retrieval (CBVR) has become an active area of research durinf the last decade. The objective of this thesis is to present applications for temporal video segmentation and video retrieval based on different mathematical models. A shot is considered as the elementary unit of a video, and is defined as a continuous sequence of frames taken from a single camera, representing an action during time. The different types of transitions that may occur in a video sequence are categorized into : abrupt and gradual transition. In this work, through statistical analysis, we segment a video into its constituent units. This is achieved by identifying transitions between adjacent shots. The first proposed algorithm aims to detect abrupt shot transitions only by measuring the similarity between consecutive frames. Given the size of the vector containing distances, it can be modeled by a log normal distribution since all the values are positive. Gradual shot transition identification is a more difficult task when compared to cut detection. Generally, a gradual transition may share similar characteristics as a dynamic segment with camera or object motion. In this work, singular value decomposition (SVD) is performed to project features from the spatial domain to the singular space. Resulting features are reduced and more refined, which makes the remaining tasks easier. The proposed system, designed for detecting both abrupt and gradual transitions, has lead to reliable performances achieving high detection rates. In addition, the acceptable computational time allows to process in real time. Once a video is partitioned into its elementary units, high-level applications can be processed, such as the key-frame extraction. Selecting representative frames from each shot to form a storyboard is considered as a static and local video summarization. In our research, we opted for a global method based on local extraction. Using refined centrist features from the singular space, we select representative frames using modified k-means clustering based on important scenes. This leads to catch pertinent frames without redoudancy in the final storyboard.

With the rapid increase in the amount of multimedia data, the researches on semantic information retrieval are facing a very challenging problem - the number of positive data instances with the target concept/object/event compared with the number of negative data instances without the target concept/object/event is much smaller, which is also called the data imbalance issue. Therefore, one of the popular topics in multimedia information processing and retrieval is data pruning, a technique that can automatically identify and prune the data instances from the training data set so that the pruned data set is able to enhance the performance of model learning, classification, and concept detection. In this paper, a novel data pruning framework which gives each transaction a weight based on multiple correspondence analysis (MCA) is proposed. These transaction weights are used as the measure for pruning the training data set. Meanwhile, the testing data set could be weighted and pruned as well so that the computational cost is reduced not only when building the model but also when applying the classifiers. Experimenting with 18 high-level concepts and the benchmark (both balanced and imbalanced) data sets from TRECVID, our proposed framework achieves promising results to enhance the concept detection performance of three well-known classifiers commonly used for concept detection.

With the rapid increase in online video services, multimedia retrieval systems are becoming increasingly important search tools to users in many different fields. In this paper we present a novel retrieval interface, "FacetBrowser", which supports the creation of multiple search "facets", to aid users carrying out complex search tasks involving multiple concepts. Each facet represents a different aspect of the search task: an assumption of this work is that search facets are best represented by sub-searches, providing users with flexibility in defining facets on the fly, rather than using pre-defined categories or metadata information as used in many other exploratory search interfaces [3, 8, 17]. Such facets can be organised into "stories" by users, facilitating users in building up sequences of related searches and material which together can be used to satisfy a work task. The interface allows more than one search to be executed and viewed simultaneously, and importantly, allows material to be reorganized between the facets, acknowledging the inter-relatedness which can often occur between search facets. The design of the FacetBrowser interface is presented, along with an experiment comparing it to a tabbed interface similar to that on modern web browsers. The results suggest that the FacetBrowser has the potential to aid users in exploring and structuring their searching effort when carrying out broad search tasks.