What is a Question? Crowdsourcing Tweet Categorization

One major way in which Amazon Mechanical Turk has been used is in the human labeling (or coding) of data, such as the relevance of search results or quality of Wikipedia articles. Recently, we used Amazon Mechanical Turk for classifying or labeling Twitter updates as questions or not. We present the design of our study and the steps that we took to address the challenges we faced in using Mechanical Turk for this labeling task. We also present our findings and some lessons learnt about the utility and effectiveness of using micro-task markets for conducting large-scale studies involving human-intelligence tasks.