Information Visualization Evaluation Using Crowdsourcing

Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research.

[1]  Martin Wattenberg,et al.  Voyagers and voyeurs: supporting asynchronous collaborative information visualization , 2007, CHI.

[2]  Aniket Kittur,et al.  CrowdScape: interactively visualizing user behavior and output , 2012, UIST.

[3]  Peter Dalsgård,et al.  Performing perception—staging aesthetics of interaction , 2008, TCHI.

[4]  Jason Dykes,et al.  Visual Analytical Approaches to Evaluate Uncertainty and Bias in Crowdsourced Crisis Information , 2012 .

[5]  Colin Ware,et al.  A Crowdsourced Approach to Colormap Assessment , 2017, EuroRV³@EuroVis.

[6]  Panagiotis G. Ipeirotis Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[7]  Marti A. Hearst,et al.  Evaluating Information Visualization via the Interplay of Heuristic Evaluation and Question-Based Scoring , 2016, CHI.

[8]  Katharina Reinecke,et al.  Predicting users' first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness , 2013, CHI.

[9]  Pingmei Xu,et al.  TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking , 2015, ArXiv.

[10]  Hari Kalva,et al.  Assessing internet video quality using crowdsourcing , 2013, CrowdMM '13.

[11]  Keiichiro Hoashi,et al.  Crowdsourcing GO: Effect of Worker Situation on Mobile Crowdsourcing Performance , 2017, CHI.

[12]  Bernice E. Rogowitz,et al.  Perceptual Organization in User-Generated Graph Layouts , 2008, IEEE Transactions on Visualization and Computer Graphics.

[13]  Tovi Grossman,et al.  The Effect of Visual Appearance on the Performance of Continuous Sliders and Visual Analogue Scales , 2016, CHI.

[14]  Isabelle Hupont,et al.  Eye Tracker in the Wild: Studying the delta between what is said and measured in a crowdsourcing experiment , 2015, CrowdMM@ACM Multimedia.

[15]  Michael Riegler,et al.  Mobile Picture Guess: A Crowdsourced Serious Game for Simulating Human Perception , 2014, SocInfo Workshops.

[16]  Kristen Grauman,et al.  CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question , 2017, CHI.

[17]  Antti Oulasvirta,et al.  Towards Perceptual Optimization of the Visual Design of Scatterplots , 2017, IEEE Transactions on Visualization and Computer Graphics.

[18]  In-Kwon Lee,et al.  Image Recoloring with Valence‐Arousal Emotion Model , 2016, Comput. Graph. Forum.

[19]  Christoph Trattner,et al.  Towards a Recommender Engine for Personalized Visualizations , 2015, UMAP.

[20]  Adam Marcus,et al.  The Effects of Sequence and Delay on Crowd Work , 2015, CHI.

[21]  Jean-Daniel Fekete,et al.  Suggested Interactivity: Seeking Perceived Affordances for Information Visualization , 2016, IEEE Transactions on Visualization and Computer Graphics.

[22]  J. Heinrich,et al.  Evaluating Viewpoint Entropy for Ribbon Representation of Protein Structure , 2016, Comput. Graph. Forum.

[23]  Gertrude Rand,et al.  Tests for the Detection and Analysis of Color-Blindness. III. The Rabkin Test , 1945 .

[24]  Robert Kosara,et al.  Arcs, Angles, or Areas: Individual Data Encodings in Pie and Donut Charts , 2016, Comput. Graph. Forum.

[25]  Jacki O'Neill,et al.  Turk-Life in India , 2014, GROUP.

[26]  M. Sheelagh T. Carpendale,et al.  Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing , 2015, Crowdsourcing and Human-Centered Experiments.

[27]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[28]  Morten Fjeld,et al.  ReTool: Interactive Microtask and Workflow Design through Demonstration , 2017, CHI.

[29]  Lane Harrison,et al.  Exploring the impact of emotion on visual judgement , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[30]  Judith Redi,et al.  Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force Crowdsourcing , 2014 .

[31]  Bongshin Lee,et al.  A Deeper Understanding of Sequence in Narrative Visualization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[32]  Gem Stapleton,et al.  Visualizing Sets with Linear Diagrams , 2015, TCHI.

[33]  Wai-Tat Fu,et al.  Enhancing reliability using peer consistency evaluation in human computation , 2013, CSCW '13.

[34]  Bernd Hamann,et al.  Progressive parallel coordinates , 2012, 2012 IEEE Pacific Visualization Symposium.

[35]  Panagiotis G. Ipeirotis,et al.  Estimating the Completion Time of Crowdsourced Tasks Using Survival Analysis Models , 2011 .

[36]  Hector Garcia-Molina,et al.  Turkalytics: analytics for human computation , 2011, WWW.

[37]  Álvaro Gomes,et al.  Crowdsourced Clustering of Computer Generated Floor Plans , 2015, CDVE.

[38]  Hanspeter Pfister,et al.  Guidelines for Effective Usage of Text Highlighting Techniques , 2016, IEEE Transactions on Visualization and Computer Graphics.

[39]  Katharina Reinecke,et al.  LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples , 2015, CSCW.

[40]  Jon Froehlich,et al.  Differences in Crowdsourced vs. Lab-based Mobile and Desktop Input Performance Data , 2017, CHI.

[41]  Yaron Singer,et al.  Pricing mechanisms for crowdsourcing markets , 2013, WWW.

[42]  Mor Naaman,et al.  Playable data: characterizing the design space of game-y infographics , 2011, CHI.

[43]  Tobias Hoßfeld,et al.  Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments , 2017, Lecture Notes in Computer Science.

[44]  Sebastian Möller,et al.  Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd , 2017, Crowdsourcing and Human-Centered Experiments.

[45]  Steven Franconeri,et al.  Perception of Average Value in Multiclass Scatterplots , 2013, IEEE Transactions on Visualization and Computer Graphics.

[46]  Michael S. Bernstein,et al.  Learning Perceptual Kernels for Visualization Design , 2014, IEEE Transactions on Visualization and Computer Graphics.

[47]  K. D. Joshi,et al.  Why Individuals Participate in Micro-task Crowdsourcing Work Environment: Revealing Crowdworkers' Perceptions , 2016, J. Assoc. Inf. Syst..

[48]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[49]  Danielle Albers Szafir,et al.  Lightness Constancy in Surface Visualization , 2016, IEEE Transactions on Visualization and Computer Graphics.

[50]  Steven Franconeri,et al.  Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries , 2018, IEEE Transactions on Visualization and Computer Graphics.

[51]  Timo Ropinski,et al.  A crowdsourcing system for integrated and reproducible evaluation in scientific visualization , 2016, 2016 IEEE Pacific Visualization Symposium (PacificVis).

[52]  Wouter Meulemans,et al.  Map LineUps: Effects of spatial structure on graphical inference , 2017, IEEE Transactions on Visualization and Computer Graphics.

[53]  Simon Breslav,et al.  Mimic: visual analytics of online micro-interactions , 2014, AVI.

[54]  Matthew O. Ward,et al.  Interactive Data Visualization - Foundations, Techniques, and Applications , 2010 .

[55]  Sean A. Munson,et al.  When (ish) is My Bus?: User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems , 2016, CHI.

[56]  Gennady L. Andrienko,et al.  Exploratory analysis of spatial and temporal data - a systematic approach , 2005 .

[57]  Bongshin Lee,et al.  Crowdsourcing for Information Visualization: Promises and Pitfalls , 2017, Crowdsourcing and Human-Centered Experiments.

[58]  Michelle A. Borkin,et al.  What Makes a Visualization Memorable? , 2013, IEEE Transactions on Visualization and Computer Graphics.

[59]  Jeffrey Heer,et al.  Selecting Semantically‐Resonant Colors for Data Visualization , 2013, Comput. Graph. Forum.

[60]  Robert Kosara,et al.  Laws of Attraction: From Perceptual Forces to Conceptual Similarity , 2010, IEEE Transactions on Visualization and Computer Graphics.

[61]  Fei Wang,et al.  PEARL: An interactive visual analytic tool for understanding personal emotion style derived from social media , 2014, 2014 IEEE Conference on Visual Analytics Science and Technology (VAST).

[62]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[63]  Lynne Baillie,et al.  Investigating Time Series Visualisations to Improve the User Experience , 2016, CHI.

[64]  Daniel McDuff,et al.  Crowdsourcing Facial Responses to Online Videos , 2012, IEEE Transactions on Affective Computing.

[65]  Alan M. MacEachren,et al.  How Maps Work - Representation, Visualization, and Design , 1995 .

[66]  John D. Kelleher,et al.  Using Icicle Trees to Encode the Hierarchical Structure of Source Code , 2016, EuroVis.

[67]  Tamara Munzner,et al.  Visualization Analysis and Design , 2014, A.K. Peters visualization series.

[68]  Lora Aroyo,et al.  First International Workshop on User Interfaces for Crowdsourcing and Human Computation , 2014, AVI.

[69]  Pat Hanrahan,et al.  Modeling how people extract color themes from images , 2013, CHI.

[70]  Pierre Dragicevic,et al.  Narratives in Crowdsourced Evaluation of Visualizations: A Double-Edged Sword? , 2017, CHI.

[71]  William Ribarsky,et al.  How locus of control influences compatibility with visualization style , 2011, 2011 IEEE Conference on Visual Analytics Science and Technology (VAST).

[72]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[73]  Mary Czerwinski,et al.  Selected Human Factors Issues in Information Visualization , 2009 .

[74]  Michael S. Bernstein,et al.  Personalization via friendsourcing , 2010, TCHI.

[75]  Lydia B. Chilton,et al.  Task search in a human computation market , 2010, HCOMP '10.

[76]  Steven Franconeri,et al.  Influencing visual judgment through affective priming , 2013, CHI.

[77]  Dana Chandler,et al.  Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2012, ArXiv.

[78]  Kwong-Sak Leung,et al.  A Survey of Crowdsourcing Systems , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[79]  Kim Marriott,et al.  HOLA: Human-like Orthogonal Network Layout , 2016, IEEE Transactions on Visualization and Computer Graphics.

[80]  Kwan-Liu Ma,et al.  A Study On Designing Effective Introductory Materials for Information Visualization , 2016, Comput. Graph. Forum.

[81]  Pierre Dragicevic,et al.  Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing , 2012, IEEE Transactions on Visualization and Computer Graphics.

[82]  Jason Dykes,et al.  Glyphs for Exploring Crowd‐sourced Subjective Survey Classification , 2014, Comput. Graph. Forum.

[83]  Jian Zhao,et al.  Annotation Graphs: A Graph-Based Visualization for Meta-Analysis of Data Based on User-Authored Annotations , 2017, IEEE Transactions on Visualization and Computer Graphics.

[84]  Stefan Dietze,et al.  A taxonomy of microtasks on the web , 2014, HT.

[85]  Yifan Hu,et al.  How to Display Group Information on Node-Link Diagrams: An Evaluation , 2014, IEEE Transactions on Visualization and Computer Graphics.

[86]  Michael Wybrow,et al.  Crowdsourcing Technology to Support Academic Research , 2015, Crowdsourcing and Human-Centered Experiments.

[87]  Charles Perin,et al.  A table!: improving temporal navigation in soccer ranking tables , 2014, CHI.

[88]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[89]  Michael Gleicher,et al.  Quantity estimation in visualizations of tagged text , 2013, CHI.

[90]  R. M. Vazquez The Checklist Manifesto How to Get Things Right , 2011 .

[91]  Nancy Argüelles,et al.  Author ' s , 2008 .

[92]  Joseph G. Davis,et al.  User interface design for crowdsourcing systems , 2014, AVI.

[93]  Klaus Mueller,et al.  Human Computation in Visualization: Using Purpose Driven Games for Robust Evaluation of Visualization Algorithms , 2012, IEEE Transactions on Visualization and Computer Graphics.

[94]  Sung-Hee Kim,et al.  Investigating the Efficacy of Crowdsourcing on Evaluating Visual Decision Supporting System , 2011 .

[95]  Michelle X. Zhou,et al.  Understand users’ comprehension and preferences for composing information visualizations , 2014, TCHI.

[96]  James P. Ahrens,et al.  ETK: An Evaluation Toolkit for Visualization User Studies , 2017, EuroVis.

[97]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[98]  Oded Nov,et al.  How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques , 2015, CHI.

[99]  Aniket Kittur,et al.  Reviewing versus doing: learning and performance in crowd assessment , 2014, CSCW.

[100]  Sarah H. Creem-Regehr,et al.  Evaluating the Impact of Binning 2D Scalar Fields , 2017, IEEE Transactions on Visualization and Computer Graphics.

[101]  Susann Fiedler,et al.  Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency , 2016, PLoS biology.

[102]  A. Glassner Interactive Storytelling: Techniques for 21st Century Fiction , 2004 .

[103]  M. Sheelagh T. Carpendale,et al.  Visualization Viewpoints , 2002 .

[104]  Isabelle Hupont,et al.  Bridging the gap between eye tracking and crowdsourcing , 2015, Electronic Imaging.

[105]  Fabio Casati,et al.  Toward effective tasks navigation in crowdsourcing , 2014, AVI.

[106]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[107]  Michael Gleicher,et al.  Task-driven evaluation of aggregation in time series visualization , 2014, CHI.

[108]  Lakshminarayanan Subramanian,et al.  Proceedings of the First ACM Symposium on Computing for Development , 2010 .

[109]  David W. McDonald,et al.  Proactive displays: Supporting awareness in fluid social environments , 2008, TCHI.

[110]  Heike Hofmann,et al.  Graphical Tests for Power Comparison of Competing Designs , 2012, IEEE Transactions on Visualization and Computer Graphics.

[111]  Steven Franconeri,et al.  Comparing averages in time series data , 2012, CHI.

[112]  Michael S. Bernstein,et al.  Analytic Methods for Optimizing Realtime Crowdsourcing , 2012, ArXiv.

[113]  Rafael Veras,et al.  Optimizing Hierarchical Visualizations with the Minimum Description Length Principle , 2017, IEEE Transactions on Visualization and Computer Graphics.

[114]  Stefan Dietze,et al.  Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys , 2015, CHI.

[115]  Steven Franconeri,et al.  Ranking Visualizations of Correlation Using Weber's Law , 2014, IEEE Transactions on Visualization and Computer Graphics.

[116]  Matthew Lease,et al.  SQUARE: A Benchmark for Research on Computing Crowd Consensus , 2013, HCOMP.

[117]  Andrew M. Webb,et al.  Using Metrics of Curation to Evaluate Information-Based Ideation , 2014, ACM Trans. Comput. Hum. Interact..

[118]  Helen C. Purchase,et al.  Experimental Human-Computer Interaction - A Practical Guide with Visual Examples , 2012 .

[119]  Krzysztof Z. Gajos,et al.  A Crowdsourced Alternative to Eye-tracking for Visualization Understanding , 2015, CHI Extended Abstracts.

[120]  Henry A. Kautz,et al.  Real-time crowd labeling for deployable activity recognition , 2013, CSCW.

[121]  Cecilia R. Aragon,et al.  Aeonium: Visual analytics to support collaborative qualitative coding , 2017, 2017 IEEE Pacific Visualization Symposium (PacificVis).

[122]  Elizabeth Gerber,et al.  From in the Class or in the Wild?: Peers Provide Better Design Feedback Than External Crowds , 2017, CHI.

[123]  Maurizio Marchese,et al.  ReLauncher: Crowdsourcing Micro-Tasks Runtime Controller , 2016, CSCW.

[124]  Katharina Reinecke,et al.  Explaining the Gap: Visualizing One's Predictions Improves Recall and Comprehension of Data , 2017, CHI.

[125]  L. Hardy,et al.  Tests for the Detection and Analysis of Color-Blindness. I. The Ishihara Test: An Evaluation , 1945 .

[126]  Vidya Setlur,et al.  Four Experiments on the Perception of Bar Charts , 2014, IEEE Transactions on Visualization and Computer Graphics.

[127]  Gabriella Kazai,et al.  Quality Management in Crowdsourcing using Gold Judges Behavior , 2016, WSDM.

[128]  Alexander Klippel,et al.  PITFALLS AND POTENTIALS OF CROWD SCIENCE: A META-ANALYSIS OF CONTEXTUAL INFLUENCES , 2015 .

[129]  Katharina Reinecke,et al.  Infographic Aesthetics: Designing for the First Impression , 2015, CHI.

[130]  Jaime Teevan,et al.  Chain Reactions: The Impact of Order on Microtask Chains , 2016, CHI.

[131]  Gang Wang,et al.  Unsupervised Clickstream Clustering for User Behavior Analysis , 2016, CHI.

[132]  Jeffrey Heer,et al.  Beyond Weber's Law: A Second Look at Ranking Visualizations of Correlation , 2016, IEEE Transactions on Visualization and Computer Graphics.

[133]  Julie Dorsey,et al.  Learning and Applying Color Styles From Feature Films , 2013, Comput. Graph. Forum.

[134]  Stefan Dietze,et al.  Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks , 2017, ACM Trans. Comput. Hum. Interact..

[135]  Paul Parsons,et al.  Assessing User Engagement in Information Visualization , 2017, CHI Extended Abstracts.

[136]  Michael S. Bernstein,et al.  Twitinfo: aggregating and visualizing microblogs for event exploration , 2011, CHI.

[137]  Gabriella Kazai,et al.  An analysis of human factors and label accuracy in crowdsourcing relevance judgments , 2013, Information Retrieval.

[138]  Krzysztof Z. Gajos,et al.  BubbleView , 2017, ACM Trans. Comput. Hum. Interact..

[139]  Lane Harrison,et al.  An Evaluation of the Impact of Visual Embellishments in Bar Charts , 2015, Comput. Graph. Forum.

[140]  Michael S. Bernstein,et al.  Break It Down: A Comparison of Macro- and Microtasks , 2015, CHI.

[141]  Panagiotis G. Ipeirotis Demographics of Mechanical Turk , 2010 .

[142]  Daniel Afergan,et al.  Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability , 2016, IEEE Transactions on Visualization and Computer Graphics.

[143]  Heli Väätäjä,et al.  Exploring augmented reality for user-generated hyperlocal news content , 2013, CHI Extended Abstracts.

[144]  Bongshin Lee,et al.  A Comparative Evaluation on Online Learning Approaches using Parallel Coordinate Visualization , 2016, CHI.

[145]  P. Ubel,et al.  Measuring Numeracy without a Math Test: Development of the Subjective Numeracy Scale , 2007, Medical decision making : an international journal of the Society for Medical Decision Making.

[146]  Xin Zhang,et al.  Intelligent Graph Layout Using Many Users' Input , 2012, IEEE Transactions on Visualization and Computer Graphics.

[147]  Anna L. Cox,et al.  Diminished Control in Crowdsourcing , 2016, ACM Trans. Comput. Hum. Interact..

[148]  Maneesh Agrawala,et al.  Generating Personalized Spatial Analogies for Distances and Areas , 2016, CHI.

[149]  Pat Hanrahan,et al.  An Empirical Model of Slope Ratio Comparisons , 2012, IEEE Transactions on Visualization and Computer Graphics.

[150]  Aniket Kittur,et al.  Instrumenting the crowd: using implicit behavioral measures to predict task performance , 2011, UIST.

[151]  Robert Kosara,et al.  Judgment Error in Pie Chart Variations , 2016, EuroVis.

[152]  Cheng Deng,et al.  HindSight: Encouraging Exploration through Direct Encoding of Personal Interaction History , 2017, IEEE Transactions on Visualization and Computer Graphics.

[153]  Alexander Toet,et al.  The Perception of Visual UncertaintyRepresentation by Non-Experts , 2014, IEEE Transactions on Visualization and Computer Graphics.

[154]  Stefano Tranquillini,et al.  Keep it simple: reward and task design in crowdsourcing , 2013, CHItaly '13.

[155]  Robert Kosara,et al.  Preconceptions and Individual Differences in Understanding Visual Metaphors , 2009, Comput. Graph. Forum.

[156]  A. Ghezzi,et al.  Crowdsourcing: A Review and Suggestions for Future Research , 2018 .

[157]  Radu Jianu,et al.  GraphUnit: Evaluating Interactive Graph Visualizations Using Crowdsourcing , 2015, Comput. Graph. Forum.

[158]  Gabriella Kazai,et al.  Worker types and personality traits in crowdsourcing relevance labels , 2011, CIKM '11.

[159]  Tobias Isenberg,et al.  Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty , 2012, IEEE Transactions on Visualization and Computer Graphics.

[160]  James Davis,et al.  Evaluating and improving the usability of Mechanical Turk for low-income workers in India , 2010, ACM DEV '10.

[161]  Jing Jin,et al.  Interactive querying of temporal data using a comic strip metaphor , 2010, 2010 IEEE Symposium on Visual Analytics Science and Technology.

[162]  Alex Endert,et al.  Finding Waldo: Learning about Users from their Interactions , 2014, IEEE Transactions on Visualization and Computer Graphics.

[163]  Alan F. Blackwell,et al.  Interaction with Uncertainty in Visualisations , 2015, EuroVis.

[164]  Björn Hartmann,et al.  Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis , 2013, IEEE Transactions on Visualization and Computer Graphics.

[165]  Jeffrey Heer,et al.  Strategies for crowdsourcing social data analysis , 2012, CHI.

[166]  Min Chen,et al.  How Ordered Is It? On the Perceptual Orderability of Visual Channels , 2016, Comput. Graph. Forum.

[167]  Jean-Daniel Fekete,et al.  Storytelling in Information Visualizations: Does it Engage Users to Explore Data? , 2015, CHI.

[168]  Hanspeter Pfister,et al.  What Makes a Visualization Memorable? , 2013, IEEE Transactions on Visualization and Computer Graphics.

[169]  Karthik Ramani,et al.  Tracing and sketching performance using blunt-tipped styli on direct-touch tablets , 2014, AVI.

[170]  Bongshin Lee,et al.  Visual Narrative Flow: Exploring Factors Shaping Data Visualization Story Reading Experiences , 2017, Comput. Graph. Forum.

[171]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[172]  Phuoc Tran-Gia,et al.  Predicting result quality in Crowdsourcing using application layer monitoring , 2014, 2014 IEEE Fifth International Conference on Communications and Electronics (ICCE).

[173]  Marco Tagliasacchi,et al.  HistoGraph -- A Visualization Tool for Collaborative Analysis of Networks from Historical Social Multimedia Collections , 2014, 2014 18th International Conference on Information Visualisation.

[174]  Eytan Adar,et al.  The impact of social information on visual judgments , 2011, CHI.

[175]  Jeffrey Heer,et al.  Regression by Eye: Estimating Trends in Bivariate Visualizations , 2017, CHI.

[176]  Jeffrey Heer,et al.  Perceptual Guidelines for Creating Rectangular Treemaps , 2010, IEEE Transactions on Visualization and Computer Graphics.