论文信息 - Some thoughts from IRIT about the scenario A of the TREC RTS 2016 and 2017 tracks

Some thoughts from IRIT about the scenario A of the TREC RTS 2016 and 2017 tracks

The TREC Real-Time Summarization (RTS) track provides a framework for evaluating systems monitoring the Twitter stream and pushing tweets to users according to given profiles. It includes metrics, files, settings and hypothesis provided by the organizers. In this work, we perform a thorough analysis of each component of the framework used for batch evaluation of scenario A in 2016 and 2017. We found some weaknesses of the metrics and took advantage of these limitations to submit our official run in the 2017 edition. The good evaluation results validate our findings. This paper also gives clear recommendations to fairly reuse the collection.

Gilles Hubert | Jose G. Moreno | Yoann Pitarch | Karen Pinel-Sauvagnat

[1] Jimmy J. Lin,et al. Assessor Differences and User Preferences in Tweet Timeline Generation , 2015, SIGIR.

[2] Gilles Hubert,et al. Everything You Always Wanted to Know About TREC RTS* (*But Were Afraid to Ask) , 2017, ArXiv.

[3] Jimmy Lin,et al. On the Reusability of "Living Labs" Test Collections: A Case Study of Real-Time Summarization , 2017, SIGIR.

[4] Jimmy J. Lin,et al. Overview of the TREC 2017 Real-Time Summarization Track , 2017, TREC.

[5] Jimmy J. Lin,et al. Overview of the TREC-2015 Microblog Track , 2015, TREC.

[6] Charles L. A. Clarke,et al. An Exploration of Evaluation Metrics for Mobile Push Notifications , 2016, SIGIR.

[7] Jimmy J. Lin,et al. Online In-Situ Interleaved Evaluation of Real-Time Push Notification Systems , 2017, SIGIR.

[8] Jimmy J. Lin,et al. Interleaved Evaluation for Retrospective Summarization and Prospective Notification on Document Streams , 2016, SIGIR.

[9] Tetsuya Sakai,et al. TREC 2014 Temporal Summarization Track Overview , 2014, TREC.