论文信息 - Problems with Kendall's tau

Problems with Kendall's tau

This poster describes a potential problem with a relatively well used measure in Information Retrieval research: Kendall's Tau rank correlation coefficient. The coefficient is best known for its use in determining the similarity of test collections when ranking sets of retrieval runs. Threshold values for the coefficient have been defined and used in a number of published studies in information retrieval. However, this poster presents results showing that basing decisions on such thresholds is not as reliableas has been assumed.

Mark Sanderson | Ian Soboroff

[1] Ellen M. Voorhees,et al. Retrieval evaluation with incomplete information , 2004, SIGIR '04.

[2] Elad Yom-Tov,et al. Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval , 2005, SIGIR '05.

[3] J M Bland,et al. Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[4] Ellen M. Voorhees,et al. Evaluation by highly relevant documents , 2001, SIGIR '01.

[5] Sung-Hyon Myaeng,et al. Characteristics of the Korean Test Collection for CLIR in NTCIR-3 , 2002, NTCIR.

[6] D. Altman,et al. STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT , 1986, The Lancet.

[7] Emine Yilmaz,et al. Estimating average precision with incomplete and imperfect judgments , 2006, CIKM '06.

[8] Mark Sanderson,et al. Forming test collections with no system pooling , 2004, SIGIR '04.

[9] M. Kendall. A NEW MEASURE OF RANK CORRELATION , 1938 .

[10] Ellen M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[11] James Allan,et al. Incremental test collections , 2005, CIKM '05.