Crowdsourcing: From Theory to Practice and Long-Term Perspectives (Dagstuhl Seminar 13361)

This report documents the program and the outcomes of Dagstuhl Seminar 13361 "Crowdsourcing: From Theory to Practice and Long-Term Perspectives". Crowdsourcing is a newly emerging service platform and business model in the Internet. In contrast to outsourcing, where a job is performed by a designated worker or employee, crowdsourcing means to outsource a job to a large, anonymous crowd of workers, the so-called human cloud, in the form of an open call. Current research in crowdsourcing addresses the following issues: crowdsourcing as a novel methodology for user-centered research; development of new services and applications based on human sensing, computation, and problem solving; engineering of improved crowdsourcing platforms including quality control mechanisms; incentive design and gamification of work; usage of crowdsourcing for professional business; theoretical frameworks for evaluation. The topic on crowdsourcing may have a huge impact on the Internet and its technical infrastructure, on society, and the future of work. In short, crowdsourcing will be a guiding paradigm and form the evolution of work in the next years. Therefore, this seminar helps coordinating research efforts in the different communities. In five presentation and discussion sessions, the diverse aspects of crowdsourcing were elaborated. The topics of the sessions covered (S1) crowdsourcing in general, (S2) industry use cases, (S3) crowdsourcing design and engineering, (S4) programming and implementing crowdsourcing, (S5) applications of crowdsourcing. The major interests of the seminar participants were then focused in four different working groups on (W1) long-term perspectives & impact on economics in five years, (W2) theory -- taxonomy and dimensions of crowdsourcing, (W3) industry use cases, (W4) crowdsourcing mechanisms and design. In parallel to this seminar, a topically related seminar on "Cloud-based Software Crowdsouring", organized by Michael N. Huhns, Wei Li, Martin Schader and Wei-Tek Tsal,(Dagstuhl Seminar 13362) took place. Therefore, a joint late night session was organized to discuss crowdsourcing with respect to ethics and its relation to social computation.

[1]  Anna Sikora,et al.  AutoTune: A Plugin-Driven Approach to the Automatic Tuning of Parallel Applications , 2012, PARA.

[2]  Gerhard Wellein,et al.  LIKWID: A Lightweight Performance-Oriented Tool Suite for x86 Multicore Environments , 2010, 2010 39th International Conference on Parallel Processing Workshops.

[3]  John M. Mellor-Crummey,et al.  Effective sampling-driven performance tools for GPU-accelerated supercomputers , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[4]  James Noble,et al.  Notes on notes on postmodern programming , 2004, SIGP.

[5]  Ken Schwaber,et al.  Agile Software Development with SCRUM , 2001 .

[6]  Alan F. Blackwell,et al.  Practice-led design and evaluation of a live visual constraint language , 2013 .

[7]  Peter Sanders,et al.  Distributed Evolutionary Graph Partitioning , 2011, ALENEX.

[8]  Georgios Karagiannis,et al.  Constrained geocast to support Cooperative Adaptive Cruise Control (CACC) merging , 2010, 2010 IEEE Vehicular Networking Conference.

[9]  Nick Collins,et al.  Live coding in laptop performance , 2003, Organised Sound.

[10]  Michael Knobloch,et al.  Energy-Aware High Performance Computing - A Survey , 2013, Adv. Comput..

[11]  Shirley Moore,et al.  Measuring Energy and Power with PAPI , 2012, 2012 41st International Conference on Parallel Processing Workshops.

[12]  Alex McLean,et al.  Visualisation of live code , 2010, EVA.

[13]  Alexey V. Vinel,et al.  3GPP LTE Versus IEEE 802.11p/WAVE: Which Technology is Able to Support Cooperative Vehicular Safety Applications? , 2012, IEEE Wireless Communications Letters.

[14]  Emmanuel Agullo,et al.  Multifrontal QR Factorization for Multicore Architectures over Runtime Systems , 2013, Euro-Par.

[15]  Peter Sanders,et al.  [Delta]-stepping: a parallelizable shortest path algorithm , 2003, J. Algorithms.

[16]  Erich Strohmaier,et al.  Apex-Map: A Global Data Access Benchmark to Analyze HPC Systems and Parallel Programming Paradigms , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[17]  Marina Papatriantafilou,et al.  Adaptive Distributed b-Matching in Overlays with Preferences , 2012, SEA.

[18]  Peter Sanders,et al.  Parallel String Sample Sort , 2013, ESA.

[19]  Moshe Y. Vardi.  Computing for humans , 2011, CACM.

[20]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[21]  Peter Sanders,et al.  Engineering a scalable high quality graph partitioner , 2009, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[22]  Mitchel Resnick,et al.  Real-time programming and the big ideas of computational literacy , 2003 .

[23]  George Ho,et al.  PAPI: A Portable Interface to Hardware Performance Counters , 1999 .

[24]  Ilja Radusch,et al.  Starting European Field Tests for CAR-2-X Communication: The DRIVE C2X Framework , 2011 .

[25]  Siegfried Benkner,et al.  Autotuning of Pattern Runtimes for Accelerated Parallel Systems , 2013, PARCO.

[26]  Cédric Augonnet,et al.  PEPPHER: Efficient and Productive Usage of Hybrid Computing Systems , 2011, IEEE Micro.

[27]  Julian Rohrhuber,et al.  Waiting and Uncertainty in Computer Music Networks , 2004, ICMC.

[28]  Takahiro Katagiri,et al.  ABCLibScript: a directive to support specification of an auto-tuning facility for numerical software , 2006, Parallel Comput..

[29]  Christian Staudt,et al.  Engineering High-Performance Community Detection Heuristics for Massive Graphs , 2013, 2013 42nd International Conference on Parallel Processing.

[30]  Andrew Rau-Chaplin,et al.  A distributed tree data structure for real-time OLAP on cloud architectures , 2013, 2013 IEEE International Conference on Big Data.

[31]  Mario Gerla,et al.  A cluster based architecture for intersection collision avoidance using heterogeneous networks , 2013, 2013 12th Annual Mediterranean Ad Hoc Networking Workshop (MED-HOC-NET).

[32]  Yi Chen,et al.  On the feasibility of UMTS-based Traffic Information Systems , 2010, Ad Hoc Networks.

[33]  Elias Yaacoub,et al.  SVC video streaming over cooperative LTE/802.11p vehicle-to-infrastructure communications , 2013, 2013 World Congress on Computer and Information Technology (WCCIT).

[34]  David Blaauw,et al.  Low Power Electronics and Design , .

[35]  Frédéric Vivien,et al.  Scheduling Tree-Shaped Task Graphs to Minimize Memory and Makespan , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[36]  Bernd Mohr,et al.  The Scalasca performance toolset architecture , 2010, Concurr. Comput. Pract. Exp..

[37]  Matthias S. Müller,et al.  The Vampir Performance Analysis Tool-Set , 2008, Parallel Tools Workshop.

[38]  Bernd Mohr,et al.  eeClust: Energy-Efficient Cluster Computing , 2010, CHPC.

[39]  Anna Sikora,et al.  How to Determine the Topology of Hierarchical Tuning Networks for Dynamic Auto-Tuning in Large-Scale Systems , 2013, ICCS.

[40]  Michael Leuschel,et al.  Translating TLA + to B for Validation with ProB , 2012, IFM.

[41]  Shin-ichi Minato,et al.  Techniques of BDD/ZDD: Brief History and Recent Activity , 2013, IEICE Trans. Inf. Syst..

[42]  Marina Papatriantafilou,et al.  A greedy algorithm for the unforecasted energy dispatch problemwith storage in smart grids , 2013, e-Energy '13.

[43]  Oscar Corcho,et al.  Semantic Sensor Network XG Final Report , 2011 .

[44]  George Bosilca,et al.  Dense Linear Algebra on Distributed Heterogeneous Hardware with a Symbolic DAG Approach , 2012 .

[45]  Samuel Aaron,et al.  Snapshots: New Possibilities for Social Digital Music-Making arising from the Storage of History , 2012, ICMC.

[46]  Thomas Hérault,et al.  Assessing the Impact of ABFT and Checkpoint Composite Strategies , 2014, 2014 IEEE International Parallel & Distributed Processing Symposium Workshops.

[47]  Julian Rohrhuber,et al.  Algorithms Today: Notes on Language Design for Just in Time Programming , 2005, ICMC.

[48]  Andrea Baiocchi,et al.  Infotainment services based on push-mode dissemination in an integrated VANET and 3G architecture , 2013, Journal of Communications and Networks.

[49]  Marina Papatriantafilou,et al.  Concurrent data structures for efficient streaming aggregation , 2014, SPAA.

[50]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[51]  Bruno Gaujal,et al.  Markov chains with discontinuous drifts have differential inclusions limits , 2012 .

[52]  Satoshi Ito,et al.  Early Experiences for Adaptation of Auto-tuning by ppOpen-AT to an Explicit Method , 2013, 2013 IEEE 7th International Symposium on Embedded Multicore Socs.

[53]  Jean-Raymond Abrial,et al.  Modeling in event-b - system and software engineering by Jean-Raymond Abrial , 2010, SOEN.

[54]  Michael J. Butler,et al.  ProB: an automated analysis toolset for the B method , 2008, International Journal on Software Tools for Technology Transfer.

[55]  Alan F. Blackwell,et al.  A Principled Approach to Developing New Languages for Live Coding , 2011, NIME.

[56]  Peter Sanders,et al.  Work-efficient matrix inversion in polylogarithmic time , 2013, SPAA.

[57]  Espen Aarseth.  Cybertext: Perspectives on Ergodic Literature , 1997 .

[58]  Alex McLean,et al.  Collaboration and learning through live coding , 2014 .

[59]  Peter Sanders,et al.  Parallel Label-Setting Multi-objective Shortest Path Search , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[60]  Martin Schulz,et al.  Beyond DVFS: A First Look at Performance under a Hardware-Enforced Power Bound , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum.

[61]  Peter Sanders,et al.  Communication efficient algorithms for fundamental big data problems , 2013, 2013 IEEE International Conference on Big Data.

[62]  Peter Sanders,et al.  Scalable distributed-memory external sorting , 2009, 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010).

[63]  Grigori Fursin.  Collective Mind: cleaning up the research and experimentation mess in computer engineering using crowdsourcing, big data and machine learning , 2013, ArXiv.

[64]  Amit P. Sheth,et al.  The SSN ontology of the W3C semantic sensor network incubator group , 2012, J. Web Semant..

[65]  Amit P. Sheth,et al.  Computing for human experience: Semantics-empowered sensors, services, and social computing on the ubiquitous Web , 2010, IEEE Internet Computing.

[66]  Kurt Mehlhorn,et al.  A Parallelization of Dijkstra's Shortest Path Algorithm , 1998, MFCS.

[67]  Lynn Miller.  Case study of customer input for a successful product , 2005, Agile Development Conference (ADC'05).

[68]  Erich Strohmaier,et al.  Architecture independent performance characterization and benchmarking for scientific applications , 2004, The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004. (MASCOTS 2004). Proceedings..

[69]  Ilja Radusch,et al.  simTD: a car-to-X system architecture for field operational tests [Topics in Automotive Networking] , 2010, IEEE Communications Magazine.

[70]  Mun Choon Chan,et al.  MobTorrent: A Framework for Mobile Internet Access from Vehicles , 2009, IEEE INFOCOM 2009.

[71]  Siegfried Benkner,et al.  HyPHI - Task Based Hybrid Execution C++ Library for the Intel Xeon Phi Coprocessor , 2013, 2013 42nd International Conference on Parallel Processing.

[72]  W. H. J. Feijen,et al.  Derivation of a termination detection algorithm for distributed computations , 1986 .

[73]  Thomas R. G. Green,et al.  Cognitive dimensions of notations , 1990 .

[74]  Robert Biddle,et al.  Collaborative Events and Shared Artefacts: Agile Interaction Designers and Developers Working Toward Common Aims , 2011, 2011 AGILE Conference.

[75]  Michael Gerndt,et al.  PERISCOPE: An Online-Based Distributed Performance Analysis Tool , 2009, Parallel Tools Workshop.

[76]  Claudio Casetti,et al.  Content downloading in vehicular networks: Bringing parked cars into the picture , 2012, 2012 IEEE 23rd International Symposium on Personal, Indoor and Mobile Radio Communications - (PIMRC).

[77]  Franz Franchetti,et al.  Highly Efficient Performance Portable Tracking of Evolving Surfaces , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium.

[78]  Amit P. Sheth,et al.  Physical-Cyber-Social Computing: An Early 21st Century Approach , 2013, IEEE Intelligent Systems.

[79]  Hannes Hartenstein,et al.  Balancing the requirements for a zero false positive/negative Forward Collision Warning , 2013, 2013 10th Annual Conference on Wireless On-demand Network Systems and Services (WONS).

[80]  Masao Fukushima,et al.  V2X Communication Technology: Field Experience and Comparative Analysis , 2013 .

[81]  Pete McBreen.  Software craftsmanship : the new imperactive , 2002 .

[82]  Dorothea Wagner,et al.  Computing Multimodal Journeys in Practice , 2013, SEA.

[83]  Falko Dressler,et al.  To crash or not to crash: Estimating its likelihood and potentials of beacon-based IVC systems , 2012, 2012 IEEE Vehicular Networking Conference (VNC).

[84]  Marion Berbineau,et al.  Multi-technology Vehicular Cooperative System Based on Software Defined Radio (SDR) , 2013, Nets4Cars/Nets4Trains.

[85]  Franz Franchetti,et al.  Autotuning a Random Walk Boolean Satisfiability Solver , 2011, ICCS.

[86]  Richard Potter,et al.  Just-in-time programming , 1993 .

[87]  Michael Knobloch,et al.  Mapping fine-grained power measurements to HPC application runtime characteristics on IBM POWER7 , 2013, Computer Science - Research and Development.

[88]  Vitaly Osipov,et al.  Efficient Parallel and External Matching , 2013, Euro-Par.

[89]  Anna Sikora,et al.  How to Scale Dynamic Tuning to Large Parallel Applications , 2013, 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum.

[90]  Alan F. Blackwell,et al.  Ten years of cognitive dimensions in visual languages and computing: Guest Editor's introduction to special issue , 2006, J. Vis. Lang. Comput..

[91]  Richard W. Vuduc,et al.  A Roofline Model of Energy , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[92]  Wolfgang Frings,et al.  Measuring power consumption on IBM Blue Gene/P , 2011, Computer Science - Research and Development.

[93]  Jean-Marc Nicod,et al.  Job Scheduling Using Successive Linear Programming Approximations of a Sparse Model , 2012, Euro-Par.