Crowdsourcing graphical perception: using mechanical turk to assess visualization design

Understanding perception is critical to effective visualization design. With its low cost and scalability, crowdsourcing presents an attractive option for evaluating the large design space of visualizations; however, it first requires validation. In this paper, we assess the viability of Amazon's Mechanical Turk as a platform for graphical perception experiments. We replicate previous studies of spatial encoding and luminance contrast and compare our results. We also conduct new experiments on rectangular area perception (as in treemaps or cartograms) and on chart size and gridline spacing. Our results demonstrate that crowdsourced perception experiments are viable and contribute new insights for visualization design. Lastly, we report cost and performance data from our experiments and distill recommendations for the design of crowdsourced studies.

[1]  W. R. Garner.  The Processing of Information and Structure , 1974 .

[2]  P. Gilmartin.  INFLUENCES OF MAP CONTEXT ON CIRCLE PERCEPTION , 1981 .

[3]  B. G. Shortridge.  Stimulus Processing Models from Psychology: Can We Use Them in Cartography? , 1982 .

[4]  W. Cleveland,et al.  Variables on Scatterplots Look More Highly Correlated When the Scales Are Increased , 1982, Science.

[5]  W. Cleveland,et al.  Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods , 1984 .

[6]  Jock D. Mackinlay,et al.  Automating the design of graphical presentations of relational information , 1986, TOGS.

[7]  David K. Simkin,et al.  An Information-Processing Analysis of Graph Perception , 1987 .

[8]  S. Lewandowsky,et al.  Discriminating strata in scatterplots , 1989 .

[9]  Jeffrey S. Torguson,et al.  Cartography: Thematic Map Design , 1990 .

[10]  S. Lewandowsky,et al.  Displaying proportions and percentages , 1991 .

[11]  Ben Shneiderman,et al.  Tree visualization with tree-maps: 2-d space-filling approach , 1992, TOGS.

[12]  Alan M. MacEachren,et al.  How Maps Work - Representation, Visualization, and Design , 1995 .

[13]  L. Tremmel.  The Visual Separability of Plotting Symbols in Scatterplots , 1995 .

[14]  Peter L. Brooks,et al.  Visualizing data , 1997 .

[15]  Martin Wattenberg,et al.  Visualizing the stock market , 1999, CHI Extended Abstracts.

[16]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[17]  Jarke J. van Wijk,et al.  Squarified Treemaps , 2000, VisSym.

[18]  Vivien Beattie,et al.  The impact of graph slope on rate of change judgments in corporate reports , 2002 .

[19]  M. Banaji,et al.  Psychological. , 2015, The journals of gerontology. Series B, Psychological sciences and social sciences.

[20]  Robert E. Kraut,et al.  Experiment 1 : Motivating Conversational Contributions Through Group Homogeneity and Individual Uniqueness , 2010 .

[21]  Olivier Chapuis,et al.  Touchstone: exploratory design of experiments , 2007, CHI.

[22]  Kori Inkpen Quinn,et al.  White rooms and morphing don't mix: setting and the evaluation of visualization techniques , 2007, CHI.

[23]  Martin Wattenberg,et al.  ManyEyes: a Site for Visualization at Internet Scale , 2007, IEEE Transactions on Visualization and Computer Graphics.

[24]  Coye Cheshire.  Selective Incentives and Generalized Information Exchange , 2007 .

[25]  Daniel J. Wigdor,et al.  Perception of elementary graphical elements in tabletop and multi-surface environments , 2007, CHI.

[26]  Pat Hanrahan,et al.  Show Me: Automatic Presentation for Visual Analysis , 2007, IEEE Transactions on Visualization and Computer Graphics.

[27]  Ron Kohavi,et al.  Practical guide to controlled experiments on the web: listen to your customers not to the hippo , 2007, KDD '07.

[28]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[29]  Maureen C. Stone,et al.  Alpha, contrast and the perception of visual metadata , 2008, CIC.

[30]  Robert Kosara,et al.  The Shaping of Information by Visual Metaphors , 2008, IEEE Transactions on Visualization and Computer Graphics.

[31]  Jeffrey Heer,et al.  Sizing the horizon: the effects of chart size and layering on the graphical perception of time series visualizations , 2009, CHI.

[32]  Matthew J. Salganik,et al.  Web-Based Experiments for the Study of Collective Social Dynamics in Cultural Markets , 2009, Top. Cogn. Sci..

[33]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[34]  T. Funkhouser,et al.  How well do line drawings depict shape? , 2009, SIGGRAPH '09.