The 2015 Text REtrieval Conference (TREC) introduced the concept of ‘Open Runs’ in response to the increasing focus on repeatability of information retrieval experiments. An Open Run is a TREC submission backed by a software repository such that the software in the repository reproduces the system that created that exact run. The ID of the repository was captured during the process of submitting the run and published as part of the metadata describing the run in the TREC proceedings. Submitting a run as an Open Run was optional: either a repository ID was provided at submission time or it was not, and further processing of the run was identical in either case. Unfortunately, this initial offering was not successful. While a healthy 79 runs were submitted as Open Runs, we could not in fact reproduce any of them. This paper explores possible reasons for the difficulties and makes suggestions for how to address the deficiencies so as to strengthen the Open Run program for TREC 2016.
[1]
Matthias Hagen,et al.
Twitter Sentiment Detection via Ensemble Classification Using Averaged Confidence Scores
,
2015,
ECIR.
[2]
Andrew Trotman,et al.
Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR)
,
2016,
SIGF.
[3]
J. Ioannidis.
Why Most Published Research Findings Are False
,
2019,
CHANCE.
[4]
Jimmy J. Lin,et al.
Reproducible Experiments on Lexical and Temporal Feedback for Tweet Search
,
2015,
ECIR.
[5]
Nicola Ferro,et al.
Rank-Biased Precision Reloaded: Reproducibility and Generalization
,
2015,
ECIR.
[6]
Mark D. Smucker,et al.
Report on the SIGIR 2010 workshop on the simulation of interaction
,
2011,
SIGF.