Using community structure detection to rank annotators when ground truth is subjective

Learning using labels provided by multiple annotators has attracted a lot of interest in the machine learning community. With the advent of crowdsourcing cheap, noisy labels are easy to obtain. This has raised the question of how to assess annotator quality. Prior work uses bayesian inference to estimate consensus labels and obtain annotator scores based on expertise; the key assumptions are that the ground truth is known and categories of labels are predefined. In applications where it is possible to have multiple ground truths, assessing annotator quality is challenging since the ranking of annotators’ is dependent on the choice of ground-truth. This paper describes a case-study in the context of annotating historic newspaper articles from the New York Public Library. The goal is to assign fine-grained categorization of articles labeled “editorial” by the Optical Character Recognition (OCR) software. The task is subjective since pre-defined categories are not available. To define the ground truth we use a Community Structure Detection (CSD) algorithm in a similarity graph formed between articles. The labels from the CSD algorithm provides the target function to be learned. Annotators labels are then viewed as related tasks that help learn this target function. The technique helps to provide insights into how to rank annotator performance using well known information retrieval metrics.