Automatic combination of multiple ranked retrieval systems

Retrieval performance can often be improved significantly by using a number of different retrieval algorithms and combining the results, in contrast to using just a single retrieval algorithm. This is because different retrieval algorithms, or retrieval experts, often emphasize different document and query features when determining relevance and therefore retrieve different sets of documents. However, it is unclear how the different experts are to be combined, in general, to yield a superior overall estimate. We propose a method by which the relevance estimates made by different experts can be automatically combined to result in superior retrieval performance. We apply the method to two expert combination tasks. The applications demonstrate that the method can identify high performance combinations of experts and also is a novel means for determining the combined effectiveness of experts.

[1]  L. Guttman.  What is Not What in Statistics , 1977 .

[2]  Jeffrey Katzer,et al.  A study of the overlap among document representations , 1983, SIGIR '83.

[3]  William H. Press,et al.  Numerical Recipes in FORTRAN - The Art of Scientific Computing, 2nd Edition , 1987 .

[4]  I. Borg.  Multidimensional similarity structure analysis , 1987 .

[5]  Paul B. Kantor,et al.  A study of information seeking and retrieving. III. Searchers, searches, and overlap , 1988, J. Am. Soc. Inf. Sci..

[6]  Michael D. Gordon.  Probabilistic and genetic algorithms in document retrieval , 1988, CACM.

[7]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. background and methodology , 1988 .

[8]  W. Bruce Croft,et al.  Term clustering of syntactic phrases , 1989, SIGIR '90.

[9]  Paul Thompson,et al.  A combination of expert opinion approach to probabilistic information retrieval, part 1: The conceptual model , 1990, Inf. Process. Manag..

[10]  Chris Buckley,et al.  A probabilistic learning approach for document indexing , 1991, TOIS.

[11]  W. Bruce Croft,et al.  Evaluation of an inference network-based retrieval model , 1991, TOIS.

[12]  Edward A. Fox,et al.  Combining Evidence from Multiple Searches , 1992, TREC.

[13]  Paul Thompson.  Description of the PRC CEO Algorithm for TREC , 1992, TREC.

[14]  Nicholas J. Belkin,et al.  The effect multiple query representations on information retrieval system performance , 1993, SIGIR.

[15]  Donna Harman,et al.  The First Text REtrieval Conference (TREC-1) , 1993 .

[16]  Donna K. Harman,et al.  Overview of the first TREC conference , 1993, SIGIR.

[17]  Yiyu Yao,et al.  Computation of term associations by a neural network , 1993, SIGIR.

[18]  Donna Harman,et al.  Overview of the First Text REtrieval Conference. , 1993, SIGIR 1993.

[19]  Brian T. Bartell,et al.  Optimizing ranking functions: a connectionist approach to adaptive information retrieval , 1994 .