After a network-based analysis of TREC results, Mizzaro and Robertson [4] found the rather unpleasant result that topic ease (i.e., the average effectiveness of the participating systems, measured with average precision) correlates with the ability of topics to predict system effectiveness (defined as topic hubness). We address this issue by: (i) performing a more detailed analysis, and (ii) using three different datasets. Our results are threefold. First, we confirm that the original result is indeed correct and general across datasets. Second, we show that, however, that result is less worrying than what might seem at first glance, since it depends on considering the least effective systems in the analysis. In other terms, easy topics discriminate most and least effective systems, but when focussing on the most effective systems only this is no longer true. Third, we also clarify what happens when using the GMAP metric.
[1]
Stephen E. Robertson.
On Smoothing Average Precision
,
2012,
ECIR.
[2]
Stefano Mizzaro,et al.
The Good, the Bad, the Difficult, and the Easy: Something Wrong with Information Retrieval Evaluation?
,
2008,
ECIR.
[3]
Stephen E. Robertson,et al.
Hits hits TREC: exploring IR evaluation results with network analysis
,
2007,
SIGIR.
[4]
Stephen E. Robertson,et al.
On GMAP: and other transformations
,
2006,
CIKM '06.
[5]
Jon Kleinberg,et al.
Authoritative sources in a hyperlinked environment
,
1999,
SODA '98.
[6]
Stephen E. Robertson,et al.
On Using Fewer Topics in Information Retrieval Evaluations
,
2013,
ICTIR.