Blind Men and Elephants: Six Approaches to TREC data

The paper reviews six recent efforts to better understand performance measurements on information retrieval (IR) systems within the framework of the Text REtrieval Conferences (TREC): analysis of variance, cluster analyses, rank correlations, beadplots, multidimensional scaling, and item response analysis. None of this work has yielded any substantial new insights. Prospects that additional work along these lines will yield more interesting results vary but are in general not promising. Some suggestions are made for paying greater attention to richer descriptions of IR system behavior but within smaller, better controlled settings.

[1]  C. L. Mallows.  NON-NULL RANKING MODELS. I , 1957 .

[2]  George A. Milliken,et al.  Analysis Messy Data ,Volume 2: Nonreplicated Experiments , 1989 .

[3]  Guillermo Oyarce,et al.  A Visualization Case Study of Feature Vector and Stemmer Effects on TREC Topic-document Subsets. , 1998 .

[4]  Donna K. Harman,et al.  Overview of the Second Text REtrieval Conference (TREC-2) , 1994, HLT.

[5]  Donna Harman,et al.  The fourth text REtrieval conference , 1996 .

[6]  D. Critchlow.  Metric Methods for Analyzing Partially Ranked Data , 1986 .

[7]  Thomas Commerford Martin,et al.  Edison, His Life and Inventions , 2001 .

[8]  W. Dixon,et al.  BMDP statistical software , 1983 .

[9]  Giles,et al.  Searching the world wide Web , 1998, Science.

[10]  James Blustein,et al.  A Statistical Analysis of the TREC-3 Data , 1995, TREC.

[11]  P. Diaconis.  Group representations in probability and statistics , 1988 .

[12]  Mark E. Rorvig,et al.  Visualization and Scaling of TREC Topic Document Sets , 1998, Inf. Process. Manag..

[13]  W. J. Dixon,et al.  BMDP Statistical Software : 1985 Printing , 1985 .

[14]  R. Brennan,et al.  Test equating : methods and practices , 1995 .

[15]  Robert J. Mislevy,et al.  BILOG 3 : item analysis and test scoring with binary logistic models , 1990 .

[16]  Gregory M. Constantine,et al.  Metric Models for Random Graphs , 1998 .

[17]  John A. Hartigan,et al.  Clustering Algorithms , 1975 .

[18]  George A. Milliken,et al.  Analysis of Messy Data, Volume II: Nonreplicated Experiments , 1989 .