Multiple-evidence techniques are touted as means to improve the effectiveness of systems. Belkin, et al. [1] examined the effects of various query representations. Fox, et al. [2] proposed several combination algorithms and found that combinations of the same types of runs (long and short queries within the vector space model) did not yield improvement and sometimes even degraded performance. He did achieve improvement over individual runs when merging different retrieval strategies (e.g., vector space and pnorm Boolean). Lee [3] further examined various combination algorithms for fusing result sets to improve effectiveness. He identified that, for multiple-evidence to improve system effectiveness, the retrieved sets must have higher relevance overlap than non-relevance overlap. Lee did not identify the exact difference needed to improve effectiveness. His results had a 125% difference in relevant and non-relevant overlap. While Lee's experiments focused on different system result sets, we focus on effective ranking strategies removing systemic differences of parsers, stemmers, phrase processing and weighting factors. We show that the improvements shown by Lee were likely produced by fusing ranking strategies less tuned than today’s measures, and current improvements are likely to be produced by systemic differences rather than ranking strategies.
[1]
Kui-Lam Kwok,et al.
TREC-7 Ad-Hoc, High Precision and Filtering Experiments using PIRCS
,
1998,
TREC.
[2]
Chris Buckley,et al.
Pivoted Document Length Normalization
,
1996,
SIGIR Forum.
[3]
Jong-Hak Lee,et al.
Analyses of multiple evidence combination
,
1997,
SIGIR '97.
[4]
Stephen E. Robertson,et al.
Okapi at TREC-4
,
1995,
TREC.
[5]
Edward A. Fox,et al.
Combination of Multiple Searches
,
1993,
TREC.
[6]
Alistair Moffat,et al.
Statistical phrases for vector-space information retrieval (poster abstract)
,
1999,
SIGIR '99.
[7]
Stephen E. Robertson,et al.
Okapi at TREC-3
,
1994,
TREC.
[8]
E. A. Fox,et al.
Combining the Evidence of Multiple Query Representations for Information Retrieval
,
1995,
Inf. Process. Manag..