VisLab: Crowdsourcing Visualization Experiments in the Wild

When creating a visualization to understand and communicate data, we face different design choices. Even though past empirical research provides foundational knowledge for visualization design, practitioners still rely on their hunches to deal with intricate trade-offs in the wild. On the other hand, researchers lack the time and resources to rigorously explore the growing design space through controlled experiments. In this work, we aim to address this two-fold problem by crowdsourcing visualization experiments. We developed VisLab, an online platform in which anyone can design and deploy experiments to evaluate their visualizations. To alleviate the complexity of experiment design and analysis, our platform provides scaffold templates and analytic dashboards. To motivate broad participation in the experiments, the platform enables anonymous participation and provides personalized performance feedback. We present use case scenarios that demonstrate the usability and usefulness of the platform in addressing the different needs of practitioners, researchers, and educators.

[1]  Margot Lovejoy,et al.  Context Providers: Conditions of Meaning in Media Arts , 2011 .

[2]  Tamara Munzner,et al.  A Multi-Level Typology of Abstract Visualization Tasks , 2013, IEEE Transactions on Visualization and Computer Graphics.

[3]  Arvind Satyanarayan,et al.  Vega-Lite: A Grammar of Interactive Graphics , 2018, IEEE Transactions on Visualization and Computer Graphics.

[4]  R McGill,et al.  Graphical Perception and Graphical Methods for Analyzing Scientific Data , 1985, Science.

[5]  Philippe Castagliola,et al.  A Comparison of the Readability of Graphs Using Node-Link and Matrix-Based Representations , 2004, IEEE Symposium on Information Visualization.

[6]  W. Cleveland,et al.  Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods , 1984 .

[7]  Bongshin Lee,et al.  Information Visualization Evaluation Using Crowdsourcing , 2018, Comput. Graph. Forum.

[8]  Paul Parsons,et al.  Assessing User Engagement in Information Visualization , 2017, CHI Extended Abstracts.

[9]  Krzysztof Z Gajos,et al.  BubbleView , 2017, ACM Trans. Comput. Hum. Interact..

[10]  Daniel A. Keim,et al.  VisGuides: A Forum for Discussing Visualization Guidelines , 2018, EuroVis.

[11]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[12]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[13]  Christophe Hurter,et al.  Data-Driven Storytelling , 2017 .

[14]  Steven Franconeri,et al.  Ranking Visualizations of Correlation Using Weber's Law , 2014, IEEE Transactions on Visualization and Computer Graphics.

[15]  Jock D. Mackinlay,et al.  Storytelling: The Next Step for Visualization , 2013, Computer.

[16]  Irene Reppa,et al.  An Empirical Study on Using Visual Embellishments in Visualization , 2012, IEEE Transactions on Visualization and Computer Graphics.

[17]  Tim Kraska,et al.  VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository , 2019, CHI.

[18]  Scott R. Klemmer,et al.  Gut Instinct: Creating Scientific Theories with Online Learners , 2017, CHI.

[19]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[20]  Jeffrey Heer,et al.  Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco , 2018, IEEE Transactions on Visualization and Computer Graphics.

[21]  Jeffrey Heer,et al.  Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation , 2008, IEEE Transactions on Visualization and Computer Graphics.

[22]  Katharina Reinecke,et al.  LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples , 2015, CSCW.

[23]  Hanspeter Pfister,et al.  Beyond Memorability: Visualization Recognition and Recall , 2016, IEEE Transactions on Visualization and Computer Graphics.

[24]  Steven Franconeri,et al.  ISOTYPE Visualization: Working Memory, Performance, and Engagement with Pictographs , 2015, CHI.

[25]  Vickie Curtis,et al.  Motivation to Participate in an Online Citizen Science Game , 2015 .

[26]  Hanspeter Pfister,et al.  What Makes a Visualization Memorable? , 2013, IEEE Transactions on Visualization and Computer Graphics.

[27]  C. Lintott,et al.  Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey , 2008, 0804.4483.