Comparing interactive information retrieval systems across sites: the TREC-6 interactive track matrix experiment

This is a case study in the design and analysis of a g-site TREC-6 experiment aimed at comparing the performance of 12 interactive information retrieval (IR) systems on a shared problem: a question-answering task, 6 statements of information need, and a collection of 210,158 articles from the Financial Times of London 1991-1994. The study discusses the application of experimental design principles and the use of a shared control IR system in addressing the problems of comparing experimental interactive IR systems across sites: isolating the effects of topics, human searchers, and other site-specific factors within an affordable design. The results confirm the dominance of the topic effect, show the searcher effect is almost as often absent as present, and indicate that for several sites the a-factor interactions are negligible. An analysis of variance found the system effect to be significant, but a multiple comparisons test found no significant pairwise differences.

[1]  Ellen M. Voorhees,et al.  The Sixth Text REtrieval Conference (TREC-6) , 2000, Inf. Process. Manag..

[2]  Christian Lenz Cesar,et al.  IBM Search UI Prototype Evaluation at the Interactive Track of TREC-6 , 1997, TREC.

[3]  Nicholas J. Belkin,et al.  Rutgers' TREC-6 Interactive Track Experience , 1997, TREC.

[4]  M. Kenward,et al.  Design and Analysis of Cross-Over Trials , 1989 .

[5]  William M. Shaw,et al.  Interactive Retrieval using IRIS: TREC-6 Experiments , 1997, TREC.

[6]  Ray R. Larson,et al.  Cheshire II at TREC 6: Interactive Probabilistic Retrieval , 1997, TREC.

[7]  William R. Hersh,et al.  A Comparison of Boolean and Natural Language Searching for the TREC-6 Interactive Task , 1997, TREC.

[8]  Ross Wilkinson,et al.  MDS TREC6 Report , 1997, TREC.

[9]  Stephen E. Robertson,et al.  On sample sizes for non-matched-pair IR experiments , 1990, Inf. Process. Manag..

[10]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[11]  Donna K. Harman,et al.  Overview of the Fifth Text REtrieval Conference (TREC-5) , 1996, TREC.

[12]  James Allan,et al.  Aspect windows, 3-D visualizations, and indirect comparisons of information retrieval systems , 1998, SIGIR '98.

[13]  Michael H. Kutner.  Applied Linear Statistical Models , 1974 .

[14]  David A. Hull.  Using statistical testing in the evaluation of retrieval experiments , 1993, SIGIR.

[15]  V. Barnett,et al.  Applied Linear Statistical Models , 1975 .

[16]  Stephen Robertson,et al.  The methodology of information retrieval experiment , 1981 .

[17]  James Allan,et al.  INQUERY Does Battle With TREC-6 , 1997, TREC.