CrowdReply: A Crowdsourced Multiple Choice Question Answering System

Crowdsourcing has applications where machine learning falls short. We describe the design, implementation, and deployment of CrowdReply, a crowdsourced multiple choice question answering (MCQA) system. We deployed an Android app to enable the audience watching “Who wants to be millionaire?” quiz-show on TV to play along on their smartphones simultaneously. The app enabled us to collect data from 1000s of users about MCQA dynamics. Our findings indicate that it is possible to aggregate the crowd’s answers to build a superplayer.

[1]  Murat Demirbas,et al.  Crowdsourcing location-based queries , 2011, 2011 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[2]  Mausam,et al.  Crowdsourcing Control: Moving Beyond Multiple Choice , 2012, UAI.

[3]  Rob Miller,et al.  VizWiz: nearly real-time answers to visual questions , 2010, UIST.

[4]  Murat Ali Bayir,et al.  Crowd-sourced sensing and collaboration using twitter , 2010, 2010 IEEE International Symposium on "A World of Wireless, Mobile and Multimedia Networks" (WoWMoM).

[5]  Javier R. Movellan,et al.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , 2009, NIPS.

[6]  Murat Demirbas,et al.  Eywa: Crowdsourced and cloudsourced omniscience , 2013, 2013 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[7]  Peng Dai,et al.  Decision-Theoretic Control of Crowd-Sourced Workflows , 2010, AAAI.

[8]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.