Diversity within the crowd

Though crowdsourcing holds great promise, many struggle with framing tasks and determining which members of the crowd should be recruited to obtain reliable output. In some cases, expert knowledge is desired but, given the time and cost constraints of the problem, may not be available. In this case, it would be beneficial to augment the expert input that is available with input from members of the general population. We believe that reduced reliance on experts will in some cases lead to acceptable performance while reducing cost and latency. In this work, we show that we are able to approach the performance of an expert group for an image labeling task, while reducing our reliance on experts by incorporating non-expert responses.