Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytelling

Data storytelling plays an important role in data workers' daily jobs since it boosts team collaboration and public communication. However, to make an appealing data story, data workers spend tremendous efforts on various tasks, including outlining and styling the story. Recently, a growing research trend has been exploring how to assist data storytelling with advanced artificial intelligence (AI). However, existing studies may focus on individual tasks in the workflow of data storytelling and do not reveal a complete picture of humans' preference for collaborating with AI. To better understand real-world needs, we interviewed eighteen data workers from both industry and academia to learn where and how they would like to collaborate with AI. Surprisingly, though the participants showed excitement about collaborating with AI, many of them also expressed reluctance and pointed out nuanced reasons. Based on their responses, we first characterize stages and tasks in the practical data storytelling workflows and the desired roles of AI. Then the preferred collaboration patterns in different tasks are identified. Next, we summarize the interviewees' reasons why and why not they would like to collaborate with AI. Finally, we provide suggestions for human-AI collaborative data storytelling to hopefully shed light on future related research.

[1]  Yun Wang,et al.  Notable: On-the-fly Assistant for Data Storytelling in Computational Notebooks , 2023, CHI.

[2]  Weiwei Cui,et al.  Erato: Cooperative Data Story Editing via Fact Interpolation , 2022, IEEE Transactions on Visualization and Computer Graphics.

[3]  Yun Wang,et al.  Towards Natural Language-Based Visualization Authoring , 2022, IEEE Transactions on Visualization and Computer Graphics.

[4]  N. Elmqvist,et al.  Roslingifier: Semi-Automated Storytelling for Animated Scatterplots , 2022, IEEE Transactions on Visualization and Computer Graphics.

[5]  Yang Shi,et al.  ColorCook: Augmenting Color Design for Dashboarding with Domain-Associated Palettes , 2022, Proc. ACM Hum. Comput. Interact..

[6]  Bongshin Lee,et al.  Investigating the Role and Interplay of Narrations and Animations in Data Videos , 2022, Comput. Graph. Forum.

[7]  K. Hosanagar,et al.  Designing Fair AI in Human Resource Management: Understanding Tensions Surrounding Algorithmic Evaluation and Envisioning Stakeholder-Centered Solutions , 2022, CHI.

[8]  Melanie K. Tory,et al.  How do you Converse with an Analytical Chatbot? Revisiting Gricean Maxims for Designing Analytical Conversational Behavior , 2022, CHI.

[9]  Justin D. Weisz,et al.  Investigating Explainability of Generative AI for Code through Scenario-based Design , 2022, IUI.

[10]  Huamin Qu,et al.  InfoColorizer: Interactive Recommendation of Color Palettes for Infographics , 2021, IEEE Transactions on Visualization and Computer Graphics.

[11]  Nan Cao,et al.  A Design Space for Applying the Freytag's Pyramid Structure to Data Stories , 2021, IEEE Transactions on Visualization and Computer Graphics.

[12]  Robert Kosara,et al.  From Jam Session to Recital: Synchronous Communication and Collaboration Around Data in Organizations , 2021, IEEE Transactions on Visualization and Computer Graphics.

[13]  Dongmei Zhang,et al.  Animated Presentation of Static Infographics with InfoMotion , 2021, Comput. Graph. Forum.

[14]  Nan Cao,et al.  AutoClips: An Automatic Approach to Video Generation from Data Facts , 2021, Comput. Graph. Forum.

[15]  Bongshin Lee,et al.  CAST: Authoring Data-Driven Chart Animations , 2021, CHI.

[16]  Bilge Mutlu,et al.  ToonNote: Improving Communication in Computational Notebooks Using Interactive Data Comics , 2021, CHI.

[17]  John T. Stasko,et al.  Data Animator: Authoring Expressive Animated Data Graphics , 2021, CHI.

[18]  Justin D. Weisz,et al.  Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML , 2021, IUI.

[19]  Alvin Cheung,et al.  Falx: Synthesis-Powered Visualization Authoring , 2021, CHI.

[20]  Krzysztof Z. Gajos,et al.  Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens , 2021, CHI.

[21]  Bongshin Lee,et al.  Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction , 2021, CHI.

[22]  Aditya G. Parameswaran,et al.  Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows , 2021, CHI.

[23]  Anamaria Crisan,et al.  Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop , 2021, CHI.

[24]  Mark O. Riedl,et al.  Expanding Explainability: Towards Social Transparency in AI systems , 2021, CHI.

[25]  Soya Park,et al.  How Much Automation Does a Data Scientist Want? , 2021, ArXiv.

[26]  Yang Shi,et al.  Calliope: Automatic Visual Data Story Generation from a Spreadsheet , 2020, IEEE Transactions on Visualization and Computer Graphics.

[27]  Weiwei Cui,et al.  Retrieve-Then-Adapt: Example-based Automatic Generation for Proportion-related Infographics , 2020, IEEE Transactions on Visualization and Computer Graphics.

[28]  Bongshin Lee,et al.  Interweaving Multimodal Interaction With Flexible Unit Visualizations for Data Exploration , 2020, IEEE Transactions on Visualization and Computer Graphics.

[29]  Jie Li,et al.  Supporting Story Synthesis: Bridging the Gap between Visual Analytics and Storytelling , 2020, IEEE Transactions on Visualization and Computer Graphics.

[30]  Daniel Cohen-Or,et al.  Exploring Visual Information Flows in Infographics , 2020, CHI.

[31]  Amy X. Zhang,et al.  How do Data Science Workers Collaborate? Roles, Workflows, and Tools , 2020, Proc. ACM Hum. Comput. Interact..

[32]  Justin D. Weisz,et al.  Trust in AutoML: exploring information needs for establishing trust in automated machine learning systems , 2020, IUI.

[33]  Yun Wang,et al.  DataShot: Automatic Generation of Fact Sheets from Tabular Data , 2020, IEEE Transactions on Visualization and Computer Graphics.

[34]  Alexander G. Gray,et al.  AutoAIViz: opening the blackbox of automated artificial intelligence with conditional parallel coordinates , 2019, IUI.

[35]  Yun Wang,et al.  Text-to-Viz: Automatic Generation of Infographics from Proportion-Related Natural Language Statements , 2019, IEEE Transactions on Visualization and Computer Graphics.

[36]  Kanit Wongsuphasawat,et al.  Goals, Process, and Challenges of Exploratory Data Analysis: An Interview Study , 2019, ArXiv.

[37]  Parikshit Ram,et al.  Human-AI Collaboration in Data Science , 2019, Proc. ACM Hum. Comput. Interact..

[38]  Weiwei Cui,et al.  Visualization Assessment: A Machine Learning Approach , 2019, 2019 IEEE Visualization Conference (VIS).

[39]  Michael J. Muller,et al.  How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation , 2019, CHI.

[40]  Kalyan Veeramachaneni,et al.  ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning , 2019, CHI.

[41]  Jeffrey Heer,et al.  Agency plus automation: Designing artificial intelligence into interactive systems , 2019, Proceedings of the National Academy of Sciences.

[42]  Marti A. Hearst,et al.  Futzing and Moseying: Interviews with Professional Data Analysts on Exploration Practices , 2019, IEEE Transactions on Visualization and Computer Graphics.

[43]  Xi Chen,et al.  InfoNice: Easy Creation of Information Graphics , 2018, CHI.

[44]  David Murray-Rust,et al.  Design Patterns for Data Comics , 2018, CHI.

[45]  Yuri Engelhardt,et al.  Narrative Design Patterns for Data-Driven Storytelling , 2018 .

[46]  Bongshin Lee,et al.  Timelines Revisited: A Design Space and Considerations for Expressive Storytelling , 2017, IEEE Transactions on Visualization and Computer Graphics.

[47]  M. Sheelagh T. Carpendale,et al.  The Emerging Genre of Data Comics , 2017, IEEE Computer Graphics and Applications.

[48]  Tobias Höllerer,et al.  ChartAccent: Annotation for data-driven storytelling , 2017, 2017 IEEE Pacific Visualization Symposium (PacificVis).

[49]  Bongshin Lee,et al.  Authoring Data-Driven Videos with DataClips , 2017, IEEE Transactions on Visualization and Computer Graphics.

[50]  Mira Dontcheva,et al.  Data-Driven Guides: Supporting Expressive Design for Information Graphics , 2017, IEEE Transactions on Visualization and Computer Graphics.

[51]  Vidya Setlur,et al.  Eviza: A Natural Language Interface for Visual Analysis , 2016, UIST.

[52]  Ece Kamar,et al.  Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence , 2016, IJCAI.

[53]  Alexander Lex,et al.  From Visual Exploration to Storytelling and Back Again , 2016, bioRxiv.

[54]  Hanspeter Pfister,et al.  Beyond Memorability: Visualization Recognition and Recall , 2016, IEEE Transactions on Visualization and Computer Graphics.

[55]  M. Sheelagh T. Carpendale,et al.  More Than Telling a Story: Transforming Data into Visually Shared Stories , 2015, IEEE Computer Graphics and Applications.

[56]  Katharina Reinecke,et al.  Infographic Aesthetics: Designing for the First Impression , 2015, CHI.

[57]  Bongshin Lee,et al.  A Deeper Understanding of Sequence in Narrative Visualization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[58]  Bongshin Lee,et al.  SketchStory: Telling More Engaging Stories with Data through Freeform Sketching , 2013, IEEE Transactions on Visualization and Computer Graphics.

[59]  Hanspeter Pfister,et al.  What Makes a Visualization Memorable? , 2013, IEEE Transactions on Visualization and Computer Graphics.

[60]  Jeffrey Heer,et al.  Enterprise Data Analysis and Visualization: An Interview Study , 2012, IEEE Transactions on Visualization and Computer Graphics.

[61]  Paul Johns,et al.  Understanding Pen and Touch Interaction for Data Exploration on Interactive Whiteboards , 2012, IEEE Transactions on Visualization and Computer Graphics.

[62]  Bruce Manciagli,et al.  Human Centered Design , 2011, Lecture Notes in Computer Science.

[63]  Jeffrey Heer,et al.  Narrative Visualization: Telling Stories with Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[64]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.