Report on the Evaluation-as-a-Service (EaaS) Expert Workshop

In this report, we summarize the outcome of the "Evaluation-as-a-Service" workshop that was held on the 5th and 6th March 2015 in Sierre, Switzerland. The objective of the meeting was to bring together initiatives that use cloud infrastructures, virtual machines, APIs (Application Programming Interface) and related projects that provide evaluation of information retrieval or machine learning tools as a service.

[1]  Benno Stein,et al.  Ousting ivory tower research: towards a web framework for providing experiments as a service , 2012, SIGIR '12.

[2]  Ioannis A. Kakadiaris,et al.  Results of the BioASQ Tasks of the Question Answering Lab at CLEF 2015 , 2015, CLEF.

[3]  Frank Hopfgartner,et al.  Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform , 2014, IIiX.

[4]  Georgios Balikas,et al.  Results of the BioASQ Track of the Question Answering Lab at CLEF 2014 , 2014, CLEF.

[5]  Axel-Cyrille Ngonga Ngomo,et al.  BioASQ: A Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering , 2012, AAAI Fall Symposium: Information Retrieval and Knowledge Discovery in Biomedical Text.

[6]  Iadh Ounis,et al.  On building a reusable Twitter corpus , 2012, SIGIR '12.

[7]  Jimmy J. Lin,et al.  Overview of the TREC-2013 Microblog Track , 2013, TREC.

[8]  Krisztian Balog,et al.  Head First: Living Labs for Ad-hoc Search Evaluation , 2014, CIKM.

[9]  Iadh Ounis,et al.  Overview of the TREC 2011 Microblog Track , 2011, TREC.

[10]  Benno Stein,et al.  Improving the Reproducibility of PAN's Shared Tasks: - Plagiarism Detection, Author Identification, and Author Profiling , 2014, CLEF.

[11]  Frank Hopfgartner,et al.  The plista dataset , 2013, NRS '13.

[12]  Allan Hanbury,et al.  Bringing the Algorithms to the Data: Cloud-Based Benchmarking for Medical Image Analysis , 2012, CLEF.

[13]  Jimmy J. Lin,et al.  Reproducible Experiments on Lexical and Temporal Feedback for Tweet Search , 2015, ECIR.

[14]  Craig MacDonald,et al.  Overview of the TREC-2012 Microblog Track , 2012, Text Retrieval Conference.

[15]  Frank Hopfgartner,et al.  Benchmarking News Recommendations in a Living Lab , 2014, CLEF.

[16]  Allan Hanbury,et al.  VISCERAL: Towards Large Data in Medical Imaging - Challenges and Directions , 2012, MCBR-CDS.