Crowdsourcing for Information Visualization: Promises and Pitfalls

Crowdsourcing offers great potential to overcome the limitations of controlled lab studies. To guide future designs of crowdsourcing-based studies for visualization, we review visualization research that has attempted to leverage crowdsourcing for empirical evaluations of visualizations. We discuss six core aspects for successful employment of crowdsourcing in empirical studies for visualization – participants, study design, study procedure, data, tasks, and metrics & measures. We then present four case studies, discussing potential mechanisms to overcome common pitfalls. This chapter will help the visualization community understand how to effectively and efficiently take advantage of the exciting potential crowdsourcing has to offer to support empirical visualization research.

[1]  Cynthia A. Brewer,et al.  Mapping Mortality: Evaluating Color Schemes for Choropleth Maps , 1997 .

[2]  Andreas Kerren,et al.  Human-Centered Visualization Environments , 2008 .

[3]  Silvia Miksch,et al.  Visualizing Sets and Set-typed Data: State-of-the-Art and Future Challenges , 2014, EuroVis.

[4]  Stephen G. Kobourov,et al.  Towards Understanding Enjoyment and Flow in Information Visualization , 2015, EuroVis.

[5]  Jean-Daniel Fekete,et al.  Interactive Random Graph Generation with Evolutionary Algorithms , 2012, Graph Drawing.

[6]  John T. Stasko,et al.  Low-level components of analytic activity in information visualization , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[7]  Michelle X. Zhou,et al.  Understand users’ comprehension and preferences for composing information visualizations , 2014, TCHI.

[8]  Olivier Chapuis,et al.  Touchstone: exploratory design of experiments , 2007, CHI.

[9]  Joseph W. Sakshaug,et al.  Nonparametric Generation of Synthetic Data for Small Geographic Areas , 2014, Privacy in Statistical Databases.

[10]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[11]  Sung-Hee Kim,et al.  Towards a Taxonomy for Evaluating User Engagement in Information Visualization , 2015 .

[12]  Irene Reppa,et al.  An Empirical Study on Using Visual Embellishments in Visualization , 2012, IEEE Transactions on Visualization and Computer Graphics.

[13]  Robert Kosara,et al.  Privacy-preserving data visualization using parallel coordinates , 2011, Electronic Imaging.

[14]  K. Bretonnel Cohen,et al.  Last Words: Amazon Mechanical Turk: Gold Mine or Coal Mine? , 2011, CL.

[15]  Joseph W. Sakshaug,et al.  Synthetic Data for Small Area Estimation , 2010, Privacy in Statistical Databases.

[16]  Stephen G. Kobourov,et al.  Comparing Node‐Link and Node‐Link‐Group Visualizations From An Enjoyment Perspective , 2016, Comput. Graph. Forum.

[17]  Daniel W. Archambault,et al.  The mental map and memorability in dynamic graphs , 2012, 2012 IEEE Pacific Visualization Symposium.

[18]  Robert Kosara,et al.  Challenges and Unsolved Problems , 2006, Human-Centered Visualization Environments.

[19]  Adam Finkelstein,et al.  How well do line drawings depict shape? , 2009, SIGGRAPH '09.

[20]  Steven Franconeri,et al.  Ranking Visualizations of Correlation Using Weber's Law , 2014, IEEE Transactions on Visualization and Computer Graphics.

[21]  Colin Ware,et al.  Visualizing graphs in three dimensions , 2008, TAP.

[22]  Nadine Moacdieh,et al.  Is “chart junk” useful? An extended examination of visual embellishment , 2014 .

[23]  Arzu Çöltekin,et al.  Evaluating the Effectiveness of Interactive Map Interface Designs: A Case Study Integrating Usability Metrics with Eye-Movement Analysis , 2009 .

[24]  Anna Monreale,et al.  Movement data anonymity through generalization , 2009, SPRINGL '09.

[25]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[26]  Steven Franconeri,et al.  ISOTYPE Visualization: Working Memory, Performance, and Engagement with Pictographs , 2015, CHI.

[27]  Klaus Mueller,et al.  SketchPadN-D: WYDIWYG Sculpting and Editing in High-Dimensional Space , 2013, IEEE Transactions on Visualization and Computer Graphics.

[28]  Niklas Elmqvist,et al.  Improving revisitation in graphs through static spatial features , 2011, Graphics Interface.

[29]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing to evaluate information visualization interfaces , 2013, CHI.

[30]  Phuoc Tran-Gia,et al.  Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com , 2011, 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing.

[31]  Andreas Kerren,et al.  Detecting Insight and Emotion in Visualization Applications with a Commercial EEG Headset , 2011, SIGRAD.

[32]  Niklas Elmqvist,et al.  Perception of Animated Node‐Link Diagrams for Dynamic Graphs , 2012, Comput. Graph. Forum.

[33]  Robert Kosara,et al.  Preconceptions and Individual Differences in Understanding Visual Metaphors , 2009, Comput. Graph. Forum.

[34]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[35]  Niklas Elmqvist,et al.  Fluid interaction for information visualization , 2011, Inf. Vis..

[36]  Sara Sara Irina Jean-Paul Christophe Maggi,et al.  How Do Display Design and User Characteristics Matter in Animations?: An Empirical Study with Air Traffic Control Displays , 2015, Cartogr. Int. J. Geogr. Inf. Geovisualization.

[37]  Thomas Grechenig,et al.  Evaluating the Effect of Style in Information Visualization , 2012, IEEE Transactions on Visualization and Computer Graphics.

[38]  H. Wainer.  A Test of Graphicacy in Children , 1980 .

[39]  M. Sheelagh T. Carpendale,et al.  A Descriptive Framework for Temporal Data Visualizations Based on Generalized Space‐Time Cubes , 2017, Comput. Graph. Forum.

[40]  Charles E. Osgood,et al.  A Cross-Cultural Study of the Affective Meanings of Color , 1973 .

[41]  Jean-Daniel Fekete,et al.  Task taxonomy for graph visualization , 2006, BELIV '06.

[42]  B. McKay,et al.  Fast generation of planar graphs , 2007 .

[43]  Radu Jianu,et al.  GraphUnit: Evaluating Interactive Graph Visualizations Using Crowdsourcing , 2015, Comput. Graph. Forum.

[44]  Kwan-Liu Ma,et al.  Stock Lamp: An Engagement-Versatile Visualization Design , 2015, CHI.

[45]  Oded Nov,et al.  How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques , 2015, CHI.

[46]  P. Newcomer,et al.  Basic Color Terms , 1971, International Journal of American Linguistics.

[47]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[48]  S. Maggi,et al.  Embodied decision making with animations , 2014 .

[49]  Wolfgang Aigner,et al.  EvalBench: A Software Library for Visualization Evaluation , 2013, Comput. Graph. Forum.

[50]  Woontack Woo,et al.  Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , 2015, CHI.

[51]  Sara Irina Fabrikant,et al.  Triangulating Eye Movement Data of Animated Displays , 2014, ET4S@GIScience.

[52]  Yifan Hu,et al.  How to Display Group Information on Node-Link Diagrams: An Evaluation , 2014, IEEE Transactions on Visualization and Computer Graphics.

[53]  Joseph W. Sakshaug,et al.  Generating synthetic data to produce public-use microdata for small geographic areas based on complex sample survey data with application to the National Health Interview Survey , 2014 .

[54]  Bernice E. Rogowitz,et al.  Perceptual Organization in User-Generated Graph Layouts , 2008, IEEE Transactions on Visualization and Computer Graphics.

[55]  Michael S. Bernstein,et al.  Learning Perceptual Kernels for Visualization Design , 2014, IEEE Transactions on Visualization and Computer Graphics.

[56]  Marcus A. Magnor,et al.  Synthetic Generation of High-Dimensional Datasets , 2011, IEEE Transactions on Visualization and Computer Graphics.

[57]  Michael Farrugia,et al.  Effective Temporal Graph Layout: A Comparative Study of Animation versus Static Display Methods , 2011, Inf. Vis..

[58]  Scott McCloud.  Understanding comics: the invisible art = Memahami komik / Scott McCloud; penerjemah S. Kinanti , 2001 .

[59]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[60]  Nicole Fruehauf.  Flow The Psychology Of Optimal Experience , 2016 .

[61]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[62]  Eliane Regina de Almeida Valiati,et al.  A taxonomy of tasks for guiding the evaluation of multidimensional visualizations , 2006, BELIV '06.

[63]  Valerie J. Bristor,et al.  Linking the Language Arts and Content Areas through Visual Technology , 1994 .

[64]  Kim Marriott,et al.  Memorability of Visual Features in Network Diagrams , 2012, IEEE Transactions on Visualization and Computer Graphics.

[65]  John Dingliana,et al.  An empirical study on the impact of edge bundling on user comprehension of graphs , 2012, AVI.

[66]  Carl Gutwin,et al.  Useful junk?: the effects of visual embellishment on comprehension and memorability of charts , 2010, CHI.

[67]  Patrick J. Bartlein,et al.  The end of the rainbow? Color schemes for improved data graphics , 2004 .

[68]  Andreas Kerren,et al.  Emotion scents: a method of representing user emotions on GUI widgets , 2013, Electronic Imaging.

[69]  Dino Pedreschi,et al.  Mobility, Data Mining and Privacy - Geographic Knowledge Discovery , 2008, Mobility, Data Mining and Privacy.

[70]  Isabelle Hupont,et al.  Bridging the gap between eye tracking and crowdsourcing , 2015, Electronic Imaging.

[71]  J. Lillo,et al.  Basic Color Terms , 2013 .

[72]  Xiaowei Ying,et al.  Graph Generation with Prescribed Feature Constraints , 2009, SDM.

[73]  M. Sheelagh T. Carpendale,et al.  Grounded evaluation of information visualizations , 2008, BELIV.

[74]  M. Sheelagh T. Carpendale,et al.  An Exploratory Study of Data Sketching for Visual Representation , 2015, Comput. Graph. Forum.

[75]  Pierre Dragicevic,et al.  Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing , 2012, IEEE Transactions on Visualization and Computer Graphics.

[76]  Stefan Dietze,et al.  Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys , 2015, CHI.

[77]  Arzu Çöltekin,et al.  Exploring the efficiency of users' visual analytics strategies based on sequence analysis of eye movement recordings , 2010, Int. J. Geogr. Inf. Sci..

[78]  Jacques Bertin,et al.  Semiologie graphique : les diagrammes les réseaux, les cartes , 1969 .

[79]  Gennady Andrienko,et al.  Privacy Issues in Geospatial Visual Analytics , 2012 .

[80]  Sara Irina Fabrikant,et al.  How users interact with a 3D geo-browser under time pressure , 2013 .

[81]  Yannis Theodoridis,et al.  On the Generation of Spatiotemporal Datasets , 1999 .

[82]  Marina Daecher,et al.  Experimental Human Computer Interaction A Practical Guide With Visual Examples , 2016 .

[83]  Arie E. Kaufman,et al.  VEEVVIE: Visual Explorer for Empirical Visualization, VR and Interaction Experiments , 2016, IEEE Transactions on Visualization and Computer Graphics.

[84]  Andreas Kerren,et al.  Network Visualization for Integrative Bioinformatics , 2014, Approaches in Integrative Bioinformatics.

[85]  Jean-Daniel Fekete,et al.  A Principled Way of Assessing Visualization Literacy , 2014, IEEE Transactions on Visualization and Computer Graphics.

[86]  Pingmei Xu,et al.  TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking , 2015, ArXiv.

[87]  Keith Andrews,et al.  A Comparative Study of Four Hierarchy Browsers using the Hierarchical Visualisation Testing Environment (HVTE) , 2007, 2007 11th International Conference Information Visualization (IV '07).

[88]  Helen C. Purchase,et al.  Which Aesthetic has the Greatest Effect on Human Understanding? , 1997, GD.

[89]  Ricardo Baeza-Yates,et al.  GraphGen: A Tool for Automatic Generation of Multipartite Graphs from Arbitrary Data , 2012, 2012 Eighth Latin American Web Congress.

[90]  Tamara Munzner,et al.  A Multi-Level Typology of Abstract Visualization Tasks , 2013, IEEE Transactions on Visualization and Computer Graphics.

[91]  Dieter W. Fellner,et al.  PCDC - On the Highway to Data - A Tool for the Fast Generation of Large Synthetic Data Sets , 2012, EuroVA@EuroVis.

[92]  M. Csíkszentmihályi.  Flow. The Psychology of Optimal Experience. New York (HarperPerennial) 1990. , 1990 .

[93]  Heidrun Schumann,et al.  Visualization of Time-Oriented Data , 2011, Human-Computer Interaction Series.

[94]  Jacki O'Neill,et al.  Being a turker , 2014, CSCW.

[95]  F. Paas,et al.  Cognitive Load Measurement as a Means to Advance Cognitive Load Theory , 2003 .

[96]  Ricardo Baeza-Yates,et al.  Automatic multi-partite graph generation from arbitrary data , 2014, J. Syst. Softw..

[97]  Daniel W. Archambault,et al.  Mental Map Preservation Helps User Orientation in Dynamic Graphs , 2012, GD.

[98]  Stephen G. Kobourov,et al.  Map‐based Visualizations Increase Recall Accuracy of Data , 2015, Comput. Graph. Forum.

[99]  Jean-Daniel Fekete,et al.  Storytelling in Information Visualizations: Does it Engage Users to Explore Data? , 2015, CHI.

[100]  Sidonie Christophe,et al.  Emotional response to map design aesthetics , 2012 .