Combining preference and absolute judgements in a crowd-sourced setting

This paper addresses the problem of obtaining gold-standard labels of objects based on subjective judgements provided by humans. Assuming each object can be associated with an underlying score, the objective of this work is to predict the underlying score efficiently and accurately based on preference and absolute judgements via experiments in a crowd-sourced setting. Unlike previous information aggregation methods which consider preference and absolute judgements independently or convert one to another in an ad-hoc way, the proposed method combines the two types of judgements directly via an unified probabilistic model. Additionally, we introduce a batch-mode active learning method which actively constructs a set of queries consisting of preference and absolute judgement tests which maximize the expected information gain at each iteration of the experiment. Experimental results show the effectiveness of the proposed method.