Privacy and Reliability in Crowdsourcing Service Delivery

Due to the anonymity and low pay of workers in crowd sourcing platforms, there may be concerns regarding reliability and privacy-preservation when using such platforms to deliver services. This paper describes a technique for jointly providing privacy and reliability through stochastic perturbation of micro task definitions and fusion rules to combine the work of several workers. A mathematical model of a crowd sourcing system using this technique is proposed and precise threshold conditions on loss of privacy when workers collude are provided. Tradeoffs between privacy, reliability, and cost are determined.

[1]  Lorrie Faith Cranor,et al.  Are your participants gaming the system?: screening mechanical turk workers , 2010, CHI.

[2]  S. Stenholm.  Information, Physics and Computation, by Marc Mézard and Andrea Montanari , 2010 .

[3]  David Alan Grier,et al.  Error Identification and Correction in Human Computation: Lessons from the WPA , 2011, Human Computation.

[4]  Aleksey Kolupaev,et al.  CAPTCHAs: Humans vs. Bots , 2008, IEEE Security & Privacy.

[5]  Shourya Roy,et al.  Beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks , 2011, Human Computation.

[6]  Devavrat Shah,et al.  Budget-optimal crowdsourcing using low-rank matrix approximations , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[7]  G. David Forney,et al.  Exponential error bounds for erasure, list, and decision feedback schemes , 1968, IEEE Trans. Inf. Theory.

[8]  Minghua Chen,et al.  Optimal Random Perturbation at Multiple Privacy Levels , 2009, Proc. VLDB Endow..

[9]  Matthew Lease,et al.  On Quality Control and Machine Learning in Crowdsourcing , 2011, Human Computation.

[10]  John C. Mitchell,et al.  How Good Are Humans at Solving CAPTCHAs? A Large Scale Evaluation , 2010, 2010 IEEE Symposium on Security and Privacy.

[11]  M. Mézard,et al.  Reconstruction on Trees and Spin Glass Transition , 2005, cond-mat/0512295.

[12]  Don Tapscott,et al.  Wikinomics: How Mass Collaboration Changes Everything , 2006 .

[13]  Devavrat Shah,et al.  Iterative Learning for Reliable Crowdsourcing Systems , 2011, NIPS.

[14]  Yu-An Sun,et al.  Human OCR: Insights from a Complex Human Computation Process , 2011 .

[15]  Ehud D. Karnin,et al.  Crowdsourcing in the Document Processing Practice - (A Short Practitioner/Visionary Paper) , 2010, ICWE Workshops.

[16]  Alexander I. Rudnicky,et al.  Using the Amazon Mechanical Turk for transcription of spoken language , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[17]  Hans Thies,et al.  Statistical Quality Control for Human-Based Electronic Services , 2010, ICSOC.

[18]  John Le,et al.  Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution , 2010 .

[19]  Matthew Lease,et al.  Improving Consensus Accuracy via Z-Score and Weighted Voting , 2011, Human Computation.

[20]  Pramod K. Varshney,et al.  Distributed Detection and Data Fusion , 1996 .

[21]  William H. Pierce.  Failure-Tolerant Computer Design , 2014 .

[22]  Hongtao Li,et al.  Modeling and evaluating information leakage caused by inferences in supply chains , 2011, Comput. Ind..

[23]  Aniket Kittur,et al.  Crowdsourcing and human computation: systems, studies and platforms , 2011, CHI EA '11.

[24]  Yochai Benkler,et al.  The wealth of networks: how social production transforms markets and freedom , 2006 .

[25]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[26]  Cynthia Dwork,et al.  Differential Privacy: A Survey of Results , 2008, TAMC.

[27]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[28]  Shipeng Yu,et al.  Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks , 2012, J. Mach. Learn. Res..

[29]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.