论文信息 - Further Analysis of Whether Batch and User Evaluations Give the Same Results with a Question-Answering Task

Further Analysis of Whether Batch and User Evaluations Give the Same Results with a Question-Answering Task

In the TREC-8 Interactive Track, our results indicated that the better performance obtained in batch searching evaluation do not translate into better performance by users in an instance recall task. This year we pursued this investigation further by performing the same experiments using the new questionanswering task adopted in the TREC-9 Interactive Track. Our results once again show that better performance in batch searching evaluation does not translate into gains for real users.

[1] Chris Buckley,et al. Pivoted Document Length Normalization , 1996, SIGIR Forum.

[2] Alistair Moffat,et al. Exploring the similarity space , 1998, SIGF.

[3] Kent L. Norman,et al. Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[4] Ellen M. Voorhees,et al. Evaluating evaluation measure stability , 2000, SIGIR '00.

[5] Stephen E. Robertson,et al. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval , 1994, SIGIR '94.

[6] R. Wolfinger,et al. Generalized linear mixed models a pseudo-likelihood approach , 1993 .

[7] Ian H. Witten,et al. Managing Gigabytes: Compressing and Indexing Documents and Images , 1999 .

[8] Andrew Turpin,et al. Do batch and user evaluations give the same results? , 2000, SIGIR '00.