When browsing large video collections, human-in-the-loop systems are essential. The system should understand the semantic information need of the user and interactively help formulate queries to satisfy that information need based on data-driven methods. Full synergy between the interacting user and the system can only be obtained when the system learns from the user interactions while providing immediate response. Doing so with dynamically changing information needs for large scale multimodal collections is a challenging task. To push the boundary of current methods, we propose to apply the state of the art in interactive multimodal learning to the complex multimodal information needs posed by the Video Browser Showdown (VBS). To that end we adapt the Exquisitor system, a highly scalable interactive learning system. Exquisitor combines semantic features extracted from visual content and text to suggest relevant media items to the user, based on user relevance feedback on previously suggested items. In this paper, we briefly describe the Exquisitor system, and its first incarnation as a VBS entrant.

What started as a field with an emphasis on optimally serving users' interactive information needs has now become dominated by methods that focus on improving the mean average precision (MAP) of a clearly defined task disconnected from its application. With the pervasiveness of the Internet and all the sensors available to derive contextual user information, it is time to bring the data and the user back together. As a field, we must consider understanding the subjective and descriptive nature of users and understanding data as equally interesting research topics that are both worthy of publication. At the 2012 ACM Second Annual International Conference on Multimedia Retrieval (ICMR) in Hong Kong, a panel took place with Marcel Worring as the moderator and the other authors of this article as the panelists. This panel discussion explored this intriguing question: Where is the user in multimedia retrieval?

We are interested in content-based tasks for visual media, especially video. Automatic assignment of semantic tags representing visual or multimodal concepts (“high-level features”) to video segments is a fundamental technology for filtering, categorization, browsing, search, and other video exploitation activities. Approaches based on using metadata, automatically recognised speech, image-image similarity, and object detection and matching all have their contributions but using semantic concepts is the one which is most scalable and has an interesting intersection with language. TRECVid is an annual benchmarking activity which has been on-going for 10 years [1]. It has focused on many tasks such as shot boundary detection, summarisation, and various forms of search. The 2012 running of TRECVid addressed 6 tasks, including Semantic INdexing (SIN) of video. The rationale for the SIN task is as follows. Given a test collection of video, a master shot reference, and a set of concept definitions, return for each concept a list of at most 2,000 shot IDs from the test collection ranked according to their likelihood of containing the concept. The data set is 291 hours of short videos with durations between 10 seconds and 3.5 minutes where there exists at least 4 positive samples of each concept. Of the 346 test concepts, a subset of 50, not known by participants at submission time, are manually judged. This task has run since the start of TRECVid and in 2012, the task is similar to previous years, except in scale. Previously, for the majority of participants the approach was to build independent concept detectors based on training a machine learning toolkit on a set of positive and negative example shots, using low-level features from the images, as described in [2]. To cater for this, a collaborative annotation of the concepts is carried out each year to build a concept bank on which to train classifiers. The TRECVid organizers also provide a set of relations between the concepts of two types: A implies B and A excludes B. Relations that can be derived by transitivity are not included. Participants are free to use the relations or not and submissions are not required to comply with them. Some of the more advanced methods from participants use the annotations of non-evaluated concepts and the ontology relations to improve the detection of the evaluated concepts.

This position paper discusses the need for considering keyword search over relational databases in the light of broader systems, where keyword search is just one of the components and which are aimed at better supporting users in their search tasks. These more complex systems call for appropriate evaluation methodologies which go beyond what is typically done today, i.e. measuring performances of components mostly in isolation or not related to the actual user needs, and, instead, able to consider the system as a whole, its constituent components, and their inter-relations with the ultimate goal of supporting actual user search tasks.

This paper reports the outcomes of a longitudinal study on the CLEF Ad Hoc track in order to assess its impact on the effectiveness of monolingual, bilingual and multilingual information access and retrieval systems. Monolingual retrieval shows a positive trend, even if the performance increase is not always steady from year to year; bilingual retrieval has demonstrated higher improvements in recent years, probably due to the better linguistic resources now available; and, multilingual retrieval exhibits constant improvement and performances comparable to bilingual (and, sometimes, even monolingual) ones.

This paper provides a discussion and analysis of methodological issues encountered during a scholarly impact and bibliometric study within the field of Computer Science (TRECVid Text Retrieval and Evaluation Conference, Video Retrieval Evaluation). The purpose of this paper is to provide a reflection and analysis of the methods used to provide useful information and guidance for those who may wish to undertake similar studies, and is of particular relevance for the academic disciplines which have publication and citation norms that may not perform well using traditional tools. Scopus and Google Scholar are discussed and a detailed comparison of the effects of different search methods and cleaning methods within and between these tools for subject and author analysis is provided. The additional database capabilities and usefulness of ‘Scopus More’ in addition to ‘Scopus General’ are discussed and evaluated. Scopus paper coverage is found to favourably compare with Google Scholar but Scholar consistently has superior performance at finding citations to those papers. These additional citations significantly increase the citation totals and also change the relative ranking of papers. Publish or Perish, a software wrapper for Google Scholar, is also examined and its limitations and some possible solutions are described. Data cleaning methods, including duplicate checks, expert domain checking of bibliographic data, and content checking of retrieved papers, are compared and their relative effects on paper and citation count discussed. Google Scholar and Scopus are also compared as tools for collecting bibliographic data for visualizations of developing trends and, owing to the comparative ease of collecting abstracts, Scopus is found far more effective.

This paper explores the impact of the MIREX (Music Information Retrieval Evaluation eXchange) evaluation initiative on scholarly research. Impact is assessed through a bibliometric evaluation of both the MIREX extended abstracts and the papers citing the MIREX results, the trial framework and methodology, or MIREX datasets. Impact is examined through number of publications and citation analysis. We further explore the primary publication venues for MIREX results, the geographic distribution of both MIREX contributors and researchers citing MIREX results, and the spread of MIREX-based research beyond the MIREX contributor teams. This analysis indicates that research in this area is highly collaborative, has achieved an international dissemination, and has grown to have a significant profile in the research literature.

This paper assesses the scholarly impact of the CLEF evaluation campaign by performing a bibliometric analysis of the citations of the CLEF 2000---2009 proceedings publications collected through Scopus and Google Scholar. Our analysis indicates a significant impact of CLEF, particularly for its well-established Adhoc, ImageCLEF, and QA labs, and for the lab/task overview publications that attract considerable interest. Moreover, initial analysis indicates that the scholarly impact of ImageCLEF is comparable to that of TRECVid.

This introductory chapter begins by explaining briefly what is intended by experimental evaluation in information retrieval in order to provide the necessary background for the rest of this volume. The major international evaluation initiatives that have adopted and implemented in various ways this common framework are then presented and their relationship to CLEF indicated. The second part of the chapter details how the experimental evaluation paradigm has been implemented in CLEF by providing a brief overview of the main activities and results obtained over the last two decades. The aim has been to build a strong multidisciplinary research community and to create a sustainable technical framework that would not simply support but would also empower both research and development and evaluation activities, while meeting and at times anticipating the demands of a rapidly evolving information society.

The research in content-based indexing and retrieval of visual information such as images and video has become one of the most populated directions in the vast area of information technologies. Social networks such as YouTube, Facebook, FileMobile, and DailyMotion host and supply facilities for accessing a tremendous amount of professional and user generated data. The areas of societal activity, such as, video protection and security, also generate thousands and thousands of terabytes of visual content. This book presents the most recent results and important trends in visual information indexing and retrieval. It is intended for young researchers, as well as, professionals looking for an algorithmic solution to a problem.

The medical tasks in ImageCLEF have been run every year from 2004–2018 and many different tasks and data sets have been used over these years. The created resources are being used by many researchers well beyond the actual evaluation campaigns and are allowing to compare the performance of many techniques on the same grounds and in a reproducible way. Many of the larger data sets are from the medical literature, as such images are easier to obtain and to share than clinical data, which was used in a few smaller ImageCLEF challenges that are specifically marked with the disease type and anatomic region. This chapter describes the main results of the various tasks over the years, including data, participants, types of tasks evaluated and also the lessons learned in organizing such tasks for the scientific community.

The VISCERAL project is building a cloud-based evaluation framework for evaluating machine learning and information retrieval algorithms on large amounts of data. Instead of downloading data and running evaluations locally, the data will be centrally available on the cloud and algorithms to be evaluated will be programmed in computing instances on the cloud, effectively bringing the algorithms to the data. This approach allows evaluations to be performed on Terabytes of data without needing to consider the logistics of moving the data or storing the data on local infrastructure. After discussing the challenges of benchmarking on big data, the design of the VISCERAL system is presented, concentrating on the components for coordinating the participants in the benchmark and managing the ground truth creation. The first two benchmarks run on the VISCERAL framework will be on segmentation and retrieval of 3D medical images.

The TREC Video Retrieval Evaluation (TRECVID) 2011 was a TREC-style video analysis and retrieval evaluation, the goal of which remains to promote progress in content-based exploitation of digital video via open, metrics-based evaluation. Over the last ten years this effort has yielded a better understanding of how systems can effectively accomplish such processing and how one can reliably benchmark their performance. TRECVID is funded by the National Institute of Standards and Technology (NIST) and other US government agencies. Many organizations and individuals worldwide contribute significant time and effort

The PROMISE network of excellence organized a two-days brainstorming workshop on 30th and 31st May 2012 in Padua, Italy, to discuss and envisage future directions and perspectives for the evaluation of information access and retrieval systems in multiple languages and multiple media. This document reports on the outcomes of this event and provides details about the six envisaged research lines: search applications; contextual evaluation; challenges in test collection design and exploitation; component-based evaluation; ongoing evaluation; and signal-aware evaluation. The ultimate goal of the PROMISE retreat is to stimulate and involve the research community along these research lines and to provide funding agencies with effective and scientifically sound ideas for coordinating and supporting information access research.

The MediaEval Multimedia Benchmark leveraged community cooperation and crowdsourcing to develop a large Internet video dataset for its Genre Tagging and Rich Speech Retrieval tasks.

The East-European Conference in Advances in Data Bases & Information Systems (ADBIS) spans 20 years of life. Here, by using simple statistic measures and bibliographic analysis, we illustrate basic characteristics and features of ADBIS, i.e. the venues, persons and countries involved. Also, its international character, its competitiveness and its status in the community are revealed. Finally, prolific authors and countries are extracted, topics of interest are deduced, collaboration issues and citing performance are illuminated.

The Cranfield paradigm has dominated information retrieval evaluation for almost 50 years. It has had a major impact on the entire domain of information retrieval since the 1960s and, compared with systematic evaluation in other domains, is very well developed and has helped very much to advance the field. This chapter summarizes some of the shortcomings in information analysis evaluation and how recent techniques help to leverage these shortcomings. The term Evaluation-as-a-Service (EaaS) was defined at a workshop that combined several approaches that do not distribute the data but use source code submission, APIs or the cloud to run evaluation campaigns. The outcomes of a white paper and the experiences gained in the VISCERAL project on cloud-based evaluation for medical imaging are explained in this paper. In the conclusions, the next steps for research infrastructures are imagined and the impact that EaaS can have in this context to make research in data science more efficient and effective.

Systematic evaluation has had a strong impact on many data analysis domains, for example, TREC and CLEF in information retrieval, ImageCLEF in image retrieval, and many challenges in conferences such as MICCAI for medical imaging and ICPR for pattern recognition. With Kaggle, a platform for machine learning challenges has also had a significant success in crowdsourcing solutions. This shows the importance to systematically evaluate algorithms and that the impact is far larger than simply evaluating a single system. Many of these challenges also showed the limits of the commonly used paradigm to prepare a data collection and tasks, distribute these and then evaluate the participants’ submissions. Extremely large datasets are cumbersome to download, while shipping hard disks containing the data becomes impractical. Confidential data can often not be shared, for example medical data, and also data from company repositories. Real-time data will never be available via static data collections as the data change over time and data preparation often takes much time. The Evaluation-as-a-Service (EaaS) paradigm tries to find solutions for many of these problems and has been applied in the VISCERAL project. In EaaS, the data are not moved but remain on a central infrastructure. In the case of VISCERAL, all data were made available in a cloud environment. Participants were provided with virtual machines on which to install their algorithms. Only a small part of the data, the training data, was visible to participants. The major part of the data, the test data, was only accessible to the organizers who ran the algorithms in the participants’ virtual machines on the test data to obtain impartial performance measures.

Systematic evaluation has an important place in information retrieval research starting with the Cranfield tests and currently with TREC (Text REtrieval Conference) and other evaluation campaigns. Such benchmarks are often mentioned to have an important impact in advancing a research field and making techniques comparable. Still, their exact impact is hard to measure. This paper aims at assessing the scholarly impact of the ImageCLEF image retrieval evaluation initiative. To this end, the papers in the proceedings published after each evaluation campaign and their citations are analysed using Scopus and Google Scholar. A significant impact of ImageCLEF could be shown through this bibliometric analysis. The differences between the employed analysis methods, each with its advantages and limitations, are also discussed.

Standard experimental datasets permit comprehensive analysis between approaches. These datasets are ubiquitous in many data science domains but uncommon in remote sensing. This paper presents the Spatial-Temporal Change in Environmental Context (STCEC) dataset, an experimental remote sensing dataset that contains changes (and non-changes) in homogeneous and heterogeneous environments, thereby, enabling researchers to test their approaches in different contexts. STCEC was tested with five pixel interpolation approaches and showed a significant difference between changes in homogeneous and heterogeneous environments. It is hoped that the dataset will be used by other researchers in future work.