Training Neural Nets to Aggregate Crowdsourced Responses

We propose a new method for aggregating crowdsourced responses, based on a deep neural network. Once trained, the aggregator network gets as input the responses of multiple participants to the same set of questions, and outputs its prediction for the correct response to each question. We empirically evaluate our approach on a dataset of responses to a standard IQ questionnaire, and show it outperforms existing state-ofthe-art methods.

[1]  Anirban Dasgupta,et al.  Aggregating crowdsourced binary ratings , 2013, WWW.

[2]  Tom Minka,et al.  How To Grade a Test Without Knowing the Answers - A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing , 2012, ICML.

[3]  Christopher M. Bishop,et al.  Students, Teachers, Exams and MOOCs: Predicting and Optimizing Attainment in Web-Based Education Using a Probabilistic Graphical Model , 2014, ECML/PKDD.

[4]  Yoram Bachrach,et al.  Hotspotting - A Probabilistic Graphical Model For Image Object Localization Through Crowdsourcing , 2013, AAAI.

[5]  Gjergji Kasneci,et al.  Crowd IQ: measuring the intelligence of crowdsourcing platforms , 2012, WebSci '12.

[6]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[7]  Gjergji Kasneci,et al.  Crowd IQ: aggregating opinions to boost performance , 2012, AAMAS.

[8]  A. Sen,et al.  Social Choice Theory , 1980 .

[9]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[10]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[11]  Zhenghao Chen,et al.  Tuned Models of Peer Assessment in MOOCs , 2013, EDM.

[12]  M. Verleysen,et al.  Classification in the Presence of Label Noise: A Survey , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Gjergji Kasneci,et al.  Bayesian Knowledge Corroboration with Logical Rules and User Feedback , 2010, ECML/PKDD.

[14]  Javier R. Movellan,et al.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , 2009, NIPS.

[15]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[16]  Lise Getoor,et al.  5 Probabilistic Relational Models , 2008 .

[17]  Charu C. Aggarwal,et al.  Mining collective intelligence in diverse groups , 2013, WWW.

[18]  J. Raven The Raven's Progressive Matrices: Change and Stability over Culture and Time , 2000, Cognitive Psychology.

[19]  A. Gibbard Manipulation of Voting Schemes: A General Result , 1973 .

[20]  A. McLennan Consequences of the Condorcet Jury Theorem for Beneficial Information Aggregation by Rational Agents , 1998, American Political Science Review.