论文信息 - Bias decreases in proportion to the number of annotators

Bias decreases in proportion to the number of annotators

The eect of the individual biases of corpus annotators on the value of reliability coecien ts is inversely proportional to the number of annotators (less one). As the number of annotators increases, the eect of their individual preferences becomes more similar to random noise. This suggests using multiple annotators as a means to control individual biases.

Massimo Poesio | Ron Artstein

[1] W. A. Scott,et al. Reliability of Content Analysis ; The Case of Nominal Scale Cording , 1955 .

[2] A. Stuart. A TEST FOR HOMOGENEITY OF THE MARGINAL DISTRIBUTIONS IN A TWO-WAY CLASSIFICATION , 1955 .

[3] Bernice W. Polemis. Nonparametric Statistics for the Behavioral Sciences , 1959 .

[4] Jacob Cohen. A Coefficient of Agreement for Nominal Scales , 1960 .

[5] Jacob Cohen,et al. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[6] J. Fleiss. Measuring nominal scale agreement among many raters. , 1971 .

[7] J J Bartko,et al. ON THE METHODS AND THEORY OF RELIABILITY , 1976, The Journal of nervous and mental disease.

[8] Klaus Krippendorff,et al. Content Analysis: An Introduction to Its Methodology , 1980 .

[9] R. Zwick,et al. Another look at interrater agreement. , 1988, Psychological bulletin.

[10] J. Carlin,et al. Bias, prevalence and kappa. , 1993, Journal of clinical epidemiology.

[11] Jean Carletta,et al. Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[12] Barbara Di Eugenio,et al. Squibs and Discussions: The Kappa Statistic: A Second Look , 2004, CL.

[13] Rebecca J. Passonneau. Computing Reliability for Coreference Annotation , 2004, LREC.

[14] P. Shrout,et al. Fleiss, Joseph L † , 2005 .

[15] Ron Artstein,et al. The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account , 2005, FCA@ACL.