CLEF 2017 Task Overview: The IR Task at the eHealth Evaluation Lab - Evaluating Retrieval Methods for Consumer Health Search

This paper provides an overview of the information retrieval (IR) Task of the CLEF 2017 eHealth Evaluation Lab. This task investigates the effectiveness of web search engines in providing access to medical information for common people that have no or little medical knowledge (health consumers). The task aims to foster advances in the development of search technologies for consumer health search by providing resources and evaluation methods to test and validate search systems. The problem considered in this year's task was to retrieve web pages to support the information needs of health consumers that are faced with a medical condition and that want to seek relevant health information online through a search engine. The task re-used the 2016 topics, to deepen the assessment pool and create a more comprehensive and reusable collection. The task had four sub-tasks: ad-hoc search, personalized search, query variations, and multilingual ad-hoc search. Seven teams participated in the task; relevance assessment is underway and assessments along with the participants results will be released at the CLEF 2017 conference. Resources for this task, including topics, assessments, evaluation scripts and participant runs are available at the task's GitHub repository: https://github.com/CLEFeHealth/CLEFeHealth2017IRtask/.

[1]  Allan Hanbury,et al.  Ranking Health Web Pages with Relevance and Understandability , 2016, SIGIR.

[2]  Allan Hanbury,et al.  Assessors Agreement: A Case Study Across Assessor Type, Payment Levels, Query Variations and Relevance Dimensions , 2016, CLEF.

[3]  Gareth J. F. Jones,et al.  Task 3 : User-centred health information retrieval : , 2014 .

[4]  Allan Hanbury,et al.  Query Variations and their Effect on Comparing Information Retrieval Systems , 2016, CIKM.

[5]  Guido Zuccon,et al.  The IR Task at the CLEF eHealth Evaluation Lab 2016: User-centred Health Information Retrieval , 2016, CLEF.

[6]  Gareth J. F. Jones,et al.  CLEF eHealth Evaluation Lab 2015, Task 2: Retrieving Information About Medical Symptoms , 2015, CLEF.

[7]  Gareth J. F. Jones,et al.  ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred Health Information Retrieval , 2014, CLEF.

[8]  Djoerd Hiemstra,et al.  MIREX: MapReduce Information Retrieval Experiments , 2010, ArXiv.

[9]  Hongfang Liu,et al.  Using Discharge Summaries to Improve Information Retrieval in Clinical Domain , 2013, CLEF.

[10]  Guido Zuccon,et al.  Building Evaluation Datasets for Consumer-Oriented Information Retrieval , 2016, LREC.

[11]  Wei Shen,et al.  An Investigation of the Eectiveness of Concept-based Approach in Medical Information Retrieval GRIUM @ CLEF2014eHealthTask 3 , 2014 .

[12]  Craig MacDonald,et al.  From Puppy to Maturity: Experiences in Developing Terrier , 2012, OSIR@SIGIR.

[13]  Allan Hanbury,et al.  The Impact of Fixed-Cost Pooling Strategies on Test Collection Bias , 2016, ICTIR.

[14]  Alistair Moffat,et al.  Rank-biased precision for measurement of retrieval effectiveness , 2008, TOIS.

[15]  Allan Hanbury.  Medical information retrieval: an instance of domain-specific search , 2012, SIGIR '12.

[16]  Guido Zuccon,et al.  Diagnose This If You Can - On the Effectiveness of Search Engines in Finding Medical Self-diagnosis Information , 2015, ECIR.

[17]  Guido Zuccon,et al.  Understandability Biased Evaluation for Information Retrieval , 2016, ECIR.

[18]  Sanna Salanterä,et al.  ShARe/CLEF eHealth Evaluation Lab 2013, Task 3: Information Retrieval to Address Patients' Questions when Reading Clinical Reports , 2013, CLEF.

[19]  Guido Zuccon,et al.  Integrating Understandability in the Evaluation of Consumer Health Search Engines , 2014, MedIR@SIGIR.

[20]  Guido Zuccon,et al.  CLEF 2017 eHealth Evaluation Lab Overview , 2017, CLEF.

[21]  Charles L. A. Clarke,et al.  Efficient and effective spam filtering and re-ranking for large web datasets , 2010, Information Retrieval.

[22]  João R. M. Palotti.  Beyond Topical Relevance: Studying Understandability and Reliability in Consumer Health Search , 2016, SIGIR.

[23]  W. Bruce Croft,et al.  Indri : A language-model based search engine for complex queries ( extended version ) , 2005 .

[24]  L. Park.  On the distribution of user persistence for rank-biased precision , 2007 .

[25]  David McDaid,et al.  Online health: untangling the web , 2010 .

[26]  Ryen W. White,et al.  Cyberchondria: Studies of the escalation of medical concerns in Web search , 2009, TOIS.

[27]  Guido Zuccon,et al.  Relevation!: An open source system for information retrieval relevance assessment , 2014, SIGIR.

[28]  Pierre Pluye,et al.  Shortcomings of health information on the Internet. , 2003, Health promotion international.