How AI Developers Overcome Communication Challenges in a Multidisciplinary Team

The development of AI applications is a multidisciplinary effort, involving multiple roles collaborating with the AI developers, an umbrella term we use to include data scientists and other AI-adjacent roles on the same team. During these collaborations, there is a knowledge mismatch between AI developers, who are skilled in data science, and external stakeholders who are typically not. This difference leads to communication gaps, and the onus falls on AI developers to explain data science concepts to their collaborators. In this paper, we report on a study including analyses of both interviews with AI developers and artifacts they produced for communication. Using the analytic lens of shared mental models, we report on the types of communication gaps that AI developers face, how AI developers communicate across disciplinary and organizational boundaries, and how they simultaneously manage issues regarding trust and expectations.

[1]  Foutse Khomh,et al.  Studying Software Engineering Patterns for Designing Machine Learning Systems , 2019, 2019 10th International Workshop on Empirical Software Engineering in Practice (IWESEP).

[2]  Gail C. Murphy,et al.  How does Machine Learning Change Software Development Practices? , 2021, IEEE Transactions on Software Engineering.

[3]  S. McComb.  Mental Model Convergence: The Shift from being an Individual to being a Team Member , 2007 .

[4]  Alexandre V. Evfimievski,et al.  AutoText: An End-to-End AutoAI Framework for Text , 2021, AAAI.

[5]  Adrian Jones,et al.  Multidisciplinary team working: collaboration and conflict. , 2006, International journal of mental health nursing.

[6]  Paul Dourish,et al.  Reading and Interpreting Ethnography , 2014, Ways of Knowing in HCI.

[7]  Gerry Stahl,et al.  Theories of Collaborative Cognition: Foundations for CSCL and CSCW Together , 2013, Computer-Supported Collaborative Learning at the Workplace.

[8]  K M Leighton,et al.  All work and no play. , 1968, Canadian Medical Association journal.

[9]  Steve Oney,et al.  How Data Scientists Use Computational Notebooks for Real-Time Collaboration , 2019, Proc. ACM Hum. Comput. Interact..

[10]  Dakuo Wang,et al.  Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study , 2020, Journal of medical Internet research.

[11]  Georgia Kougka,et al.  The many faces of data-centric workflow optimization: a survey , 2017, International Journal of Data Science and Analytics.

[12]  Matthias Scheutz,et al.  A Framework for Developing and Using Shared Mental Models in Human-Agent Teams , 2017 .

[13]  Anbang Xu,et al.  A New Chatbot for Customer Service on Social Media , 2017, CHI.

[14]  S. Mohammed,et al.  Metaphor No More: A 15-Year Review of the Team Mental Model Construct , 2010 .

[15]  Michael J. Muller,et al.  Information Curators in an Enterprise File-Sharing Service , 2009, ECSCW.

[16]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[17]  H. H. Clark.  Context and Common Ground , 2006 .

[18]  Christoph Treude,et al.  Empirical Studies on Collaboration in Software Development: A Systematic Literature Review , 2009 .

[19]  Hironori Washizaki,et al.  Preliminary Systematic Literature Review of Machine Learning System Development Process , 2019, ArXiv.

[20]  Philip J. Guo,et al.  Software Developers Learning Machine Learning: Motivations, Hurdles, and Desires , 2019, 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[21]  Kush R. Varshney,et al.  Increasing Trust in AI Services through Supplier's Declarations of Conformity , 2018, IBM J. Res. Dev..

[22]  J. R. Larson,et al.  Discussion of shared and unshared information in decision-making groups , 1994 .

[23]  John Zimmerman,et al.  Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes , 2019, CHI.

[24]  Amy X. Zhang,et al.  How do Data Science Workers Collaborate? Roles, Workflows, and Tools , 2020, Proc. ACM Hum. Comput. Interact..

[25]  Marian G. Williams,et al.  Translation between software designers and users , 1993, CACM.

[26]  P. Kirschner,et al.  Team learning: building shared mental models , 2011 .

[27]  Drew Paine,et al.  From The Matrix to a Model of Coordinated Action (MoCA): A Conceptual Framework of and for CSCW , 2015, CSCW.

[28]  Erick Oduor,et al.  AutoDS: Towards Human-Centered Automation of Data Science , 2021, CHI.

[29]  Kathleen H. Pine,et al.  The Politics of Measurement and Action , 2015, CHI.

[30]  Steve Oney,et al.  Callisto: Capturing the "Why" by Connecting Conversations with Computational Narratives , 2020, CHI.

[31]  Michael J. Muller,et al.  How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation , 2019, CHI.

[32]  Kush R. Varshney,et al.  Experiences with Improving the Transparency of AI Models and Services , 2019, CHI Extended Abstracts.

[33]  E. Salas,et al.  Shared mental models in expert team decision making. , 1993 .

[34]  Inioluwa Deborah Raji,et al.  Model Cards for Model Reporting , 2018, FAT.

[35]  Mark Warschauer,et al.  Same benefits, different communication patterns: Comparing Children's reading with a conversational agent vs. a human partner , 2021, Comput. Educ..

[36]  B. L. William Wong,et al.  An interactive human centered data science approach towards crime pattern analysis , 2019, Inf. Process. Manag..

[37]  Michael J. Muller,et al.  Human-Centered Study of Data Science Work Practices , 2019, CHI Extended Abstracts.

[38]  Stacie Petter,et al.  Understanding agile software development practices using shared mental models theory , 2014, Inf. Softw. Technol..

[39]  John R. Austin.  Transactive memory in organizational groups: the effects of content, consensus, specialization, and accuracy on group performance. , 2003, The Journal of applied psychology.

[40]  Aditya G. Parameswaran,et al.  DataHub: Collaborative Data Science & Dataset Version Management at Scale , 2014, CIDR.

[41]  Miryung Kim,et al.  The Emerging Role of Data Scientists on Software Development Teams , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[42]  Feng Tian,et al.  “Brilliant AI Doctor” in Rural Clinics: Challenges in AI-Powered Clinical Decision Support System Deployment , 2021, CHI.

[43]  Parikshit Ram,et al.  Human-AI Collaboration in Data Science , 2019, Proc. ACM Hum. Comput. Interact..

[44]  Li-Te Cheng,et al.  Shared waypoints and social tagging to support collaboration in software development , 2006, CSCW '06.

[45]  Rachel K. E. Bellamy,et al.  Trials and tribulations of developers of intelligent systems: A field study , 2016, 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[46]  Wil M.P. van der Aalst.  Data Scientist: The Engineer of the Future , 2014, I-ESA.

[47]  Clayton J. Hutto,et al.  Developing a Research Agenda for Human-Centered Data Science , 2016, CSCW Companion.

[48]  Martin C. Rinard,et al.  AL: autogenerating supervised learning programs , 2019, Proc. ACM Program. Lang..

[49]  Soya Park,et al.  How Much Automation Does a Data Scientist Want? , 2021, ArXiv.

[50]  Aaron Halfaker,et al.  Mapping Out Human-Centered Data Science: Methods, Approaches, and Best Practices , 2020, GROUP.

[51]  Leif Singer,et al.  How Social and Communication Channels Shape and Challenge a Participatory Culture in Software Development , 2017, IEEE Transactions on Software Engineering.

[52]  Andrea Forte,et al.  Reliability and Inter-rater Reliability in Qualitative Research , 2019, Proc. ACM Hum. Comput. Interact..

[53]  Eric Horvitz,et al.  Learning to Complement Humans , 2020, IJCAI.

[54]  Claudio S. Pinhanez,et al.  Machine Teaching by Domain Experts: Towards More Humane, Inclusive, and Intelligent Machine Learning Systems , 2019, ArXiv.

[55]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[56]  Brad A. Myers,et al.  The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool , 2018, CHI.

[57]  Mary Beth Rosson,et al.  Survey on user interface programming , 1992, CHI.

[58]  Michael Twidale,et al.  Supporting multidisciplinary collaboration: requirements from novel HCI education , 2007, CHI.

[59]  James D. Hollan,et al.  Exploration and Explanation in Computational Notebooks , 2018, CHI.

[60]  Jan Bosch,et al.  Software Engineering Challenges of Deep Learning , 2018, 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA).

[61]  Soya Park,et al.  Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models , 2021, IUI.

[62]  Tristan E. Johnson,et al.  TOWARD UNDERSTANDING THE DYNAMIC RELATIONSHIP BETWEEN TEAM AND TASK SHARED MENTAL MODELS AS DETERMINANTS OF TEAM AND INDIVIDUAL PERFORMANCES , 2012 .

[63]  Leslie A. DeChurch,et al.  Mental model metrics and team adaptability: A multi-facet multi-method examination , 2010 .

[64]  Christine T. Wolf,et al.  ModelLens: An Interactive System to Support the Model Improvement Practices of Data Science Teams , 2019, CSCW Companion.

[65]  Ivica Crnkovic,et al.  A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation , 2019, XP.