Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation

This paper presents an overview of the ImageCLEF 2018 evaluation campaign, an event that was organized as part of the CLEF (Conference and Labs of the Evaluation Forum) Labs 2018. ImageCLEF is an ongoing initiative (it started in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval with the aim of providing information access to collections of images in various usage scenarios and domains. In 2018, the 16th edition of ImageCLEF ran three main tasks and a pilot task: (1) a caption prediction task that aims at predicting the caption of a figure from the biomedical literature based only on the figure image; (2) a tuberculosis task that aims at detecting the tuberculosis type, severity and drug resistance from CT (Computed Tomography) volumes of the lung; (3) a LifeLog task (videos, images and other sources) about daily activities understanding and moment retrieval, and (4) a pilot task on visual question answering where systems are tasked with answering medical questions. The strong participation, with over 100 research groups registering and 31 submitting results for the tasks, shows an increasing interest in this benchmarking campaign.

[1]  Martha Palmer,et al.  Verb Semantics and Lexical Selection , 1994, ACL.

[2]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[3]  Mark Sanderson,et al.  The CLEF 2004 Cross-Language Image Retrieval Track , 2004, CLEF.

[4]  Marina Bosch,et al.  ImageCLEF, Experimental Evaluation in Visual Information Retrieval , 2010 .

[5]  Henning Müller,et al.  Assessing the Scholarly Impact of ImageCLEF , 2011, CLEF.

[6]  Allan Hanbury,et al.  Bringing the Algorithms to the Data: Cloud-Based Benchmarking for Medical Image Analysis , 2012, CLEF.

[7]  Axel-Cyrille Ngonga Ngomo,et al.  BioASQ: A Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering , 2012, AAAI Fall Symposium: Information Retrieval and Knowledge Discovery in Biomedical Text.

[8]  Erhard Rahm,et al.  The Scholarly Impact of CLEF (2000-2009) , 2013, CLEF.

[9]  José Francisco Aldana Montes,et al.  General Overview of ImageCLEF at the CLEF 2015 Labs , 2015, CLEF.

[10]  Henning Müller,et al.  Efficient and fully automatic segmentation of the lungs in CT volumes , 2015, VISCERAL Challenge@ISBI.

[11]  Jimmy J. Lin,et al.  Evaluation-as-a-Service: Overview and Outlook , 2015, ArXiv.

[12]  Margaret Mitchell,et al.  VQA: Visual Question Answering , 2015, International Journal of Computer Vision.

[13]  Henning Müller,et al.  Evaluating performance of biomedical image retrieval systems - An overview of the medical image retrieval task at ImageCLEF 2004-2013 , 2015, Comput. Medical Imaging Graph..

[14]  Emmanuel Dellandréa,et al.  General Overview of ImageCLEF at the CLEF 2016 Labs , 2016, CLEF.

[15]  Luca Soldaini.  QuickUMLS: a fast, unsupervised approach for medical concept extraction , 2016 .

[16]  Aditya Sharma,et al.  Estimating the future burden of multidrug-resistant and extensively drug-resistant tuberculosis in India, the Philippines, Russia, and South Africa: a mathematical modelling study. , 2017, The Lancet. Infectious diseases.

[17]  Michael Riegler,et al.  Overview of ImageCLEF 2017: Information Extraction from Images , 2017, CLEF.

[18]  Rami Albatal,et al.  Overview of NTCIR-13 Lifelog-2 Task , 2017, NTCIR.

[19]  Michael Riegler,et al.  Overview of ImageCLEFlifelog 2017: Lifelog Retrieval and Summarization , 2017, CLEF.

[20]  Guido Zuccon,et al.  CLEF 2017 eHealth Evaluation Lab Overview , 2017, CLEF.

[21]  Alexander M. Rush,et al.  Adversarially Regularized Autoencoders for Generating Discrete Structures , 2017, ArXiv.

[22]  Hervé Glotin,et al.  LifeCLEF 2017 Lab Overview: Multimedia Species Identification Challenges , 2017, CLEF.

[23]  Working Notes Proceedings of the MediaEval 2017 Workshop co-located with the Conference and Labs of the Evaluation Forum (CLEF 2017), Dublin, Ireland, September 13-15, 2017 , 2017, MediaEval.

[24]  Arzucan Özgür,et al.  BIOSSES: a semantic sentence similarity estimation system for the biomedical domain , 2017, Bioinform..

[25]  Henning Müller,et al.  Overview of ImageCLEFcaption 2017 - Image Caption Prediction and Concept Detection for Biomedical Images , 2017, CLEF.

[26]  Feifan Liu,et al.  UMass at ImageCLEF Caption Prediction 2018 Task , 2018, CLEF.

[27]  Vinh-Tiep Nguyen,et al.  Lifelog Moment Retrieval with Visual Concept Fusion and Text-based Query Expansion , 2018, CLEF.

[28]  Henning Müller,et al.  Overview of ImageCLEFtuberculosis 2018 - Detecting Multi-Drug Resistance, Classifying Tuberculosis Types and Assessing Severity Scores , 2018, CLEF.

[29]  Theodore Kalamboukis,et al.  IPL at ImageCLEF 2018: A kNN-based Concept Detection Approach , 2018, CLEF.

[30]  A. Gabrielian,et al.  Radiological signs associated with pulmonary multi-drug resistant tuberculosis: an analysis of published evidences. , 2018, Quantitative imaging in medicine and surgery.

[31]  Michael Riegler,et al.  An Interactive Lifelog Retrieval System for Activities of Daily Living Understanding , 2018, CLEF.

[32]  Henning Müller,et al.  Overview of ImageCLEF 2018 Medical Domain Visual Question Answering Task , 2018, CLEF.

[33]  Minh-Son Dao,et al.  Leveraging Content and Context in Understanding Activities of Daily Living , 2018, CLEF.

[34]  Henning Müller,et al.  Deep Multimodal Classification of Image Types in Biomedical Journal Figures , 2018, CLEF.

[35]  Yu Zhang,et al.  ImageSem at ImageCLEF 2018 Caption Task: Image Retrieval and Transfer Learning , 2018, CLEF.

[36]  Bogdan Ionescu,et al.  Multimedia Lab @ ImageCLEF 2018 Lifelog Moment Retrieval Task , 2018, CLEF.

[37]  Marie-Francine Moens,et al.  Generating Text from Images in a Smooth Representation Space , 2018, CLEF.

[38]  Michael Riegler,et al.  Overview of ImageCLEFlifelog 2018: Daily Living Understanding and Lifelog Moment Retrieval , 2018, CLEF.

[39]  Hsin-Hsi Chen,et al.  Visual Concept Selection with Textual Knowledge for Understanding Activities of Daily Living and Life Moment Retrieval , 2018, CLEF.

[40]  Mahmudur Rahman.  A Cross Modal Deep Learning Based Approach for Caption Prediction and Concept Detection by CS Morgan State , 2018, CLEF.

[41]  Carlos Costa,et al.  Feature Learning with Adversarial Networks for Concept Detection in Medical Images: UA.PT Bioinformatics at ImageCLEF 2018 , 2018, CLEF.

[42]  Henning Müller,et al.  Overview of the ImageCLEF 2018 Caption Prediction Tasks , 2018, CLEF.