论文信息 - Replicability and Reproducibility of Automatic Routing Runs

Replicability and Reproducibility of Automatic Routing Runs

This paper reports our participation in CENTRE@CLEF19. We focus on reimplementing submissions by Grossman and Cormack to the TREC 2017 Common Core Track. Our contributions are twofold. Reimplementations are used to study the replicability as well as the reproducibility of WCRobust04 and WCRobust0405. Our results show that the replicability and reproducibility of transferring relevance judgments across different corpora are limited. It is not possible to replicate or reproduce the baseline. However, improvements in evaluation measures by enriching training data are achievable. Further experiments examine general relevance transfer and the augmentation of tfidf-features.

Timo Breuer | Philipp Schaer

[1] Tetsuya Sakai,et al. Centre@clef 2019 , 2019, ECIR.

[2] Craig MacDonald,et al. Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge , 2016, ECIR.

[3] Ellen M. Voorhees,et al. The TREC robust retrieval track , 2005, SIGF.

[4] Gaël Varoquaux,et al. Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[5] Andrew Trotman,et al. Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR) , 2016, SIGF.

[6] Ellen M. Voorhees,et al. Overview of the TREC 2004 Robust Retrieval Track , 2004 .

[7] Jimmy Lin,et al. Simple Techniques for Cross-Collection Relevance Feedback , 2019, ECIR.

[8] Ellen M. Voorhees,et al. Overview of the TREC 2004 Robust Track. , 2004 .

[9] Maura R. Grossman,et al. MRG_UWaterloo and WaterlooCormack Participation in the TREC 2017 Common Core Track , 2017, TREC.

[10] Tetsuya Sakai,et al. Overview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm , 2018, CLEF.