Discovering the Sweet Spot of Human-Computer Configurations

Interactive intelligent systems, i.e., interactive systems that employ AI technologies, are currently present in many parts of our social, public and political life. An issue reoccurring often in the development of these systems is the question regarding the level of appropriate human and computer contributions. Engineers and designers lack a way of systematically defining and delimiting possible options for designing such systems in terms of levels of automation. In this paper, we propose, apply and reflect on a method for human-computer configuration design. It supports the systematic investigation of the design space for developing an interactive intelligent system. We illustrate our method with a use case in the context of collaborative ideation. Here, we developed a tool for information extraction from idea content. A challenge was to find the right level of algorithmic support, whereby the quality of the information extraction should be as high as possible, but, at the same time, the human effort should be low. Such contradicting goals are often an issue in system development; thus, our method proposed helped us to conceptualize and explore the design space. Based on a critical reflection on our method application, we want to offer a complementary perspective to the value-centered design of interactive intelligent systems. Our overarching goal is to contribute to the design of so-called hybrid systems where humans and computers are partners.

[1]  Ross A. Knepper,et al.  Implicit Communication of Actionable Information in Human-AI teams , 2019, CHI.

[2]  Jonathan Grudin,et al.  Chatbots, Humbots, and the Quest for Artificial General Intelligence , 2019, CHI.

[3]  Emily M. Bender,et al.  Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science , 2018, TACL.

[4]  Aaron Halfaker,et al.  Value-Sensitive Algorithm Design , 2018, Proc. ACM Hum. Comput. Interact..

[5]  Inioluwa Deborah Raji,et al.  Model Cards for Model Reporting , 2018, FAT.

[6]  Ahmed Hosny,et al.  The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards , 2018, Data Protection and Privacy.

[7]  Claudia Müller-Birn,et al.  Concept Validation during Collaborative Ideation and Its Effect on Ideation Outcome , 2018, CHI Extended Abstracts.

[8]  Tyler H. Shaw,et al.  From ‘automation’ to ‘autonomy’: the importance of trust repair in human–machine interaction , 2018, Ergonomics.

[9]  Timnit Gebru,et al.  Datasheets for datasets , 2018, Commun. ACM.

[10]  Emily M. Bender,et al.  Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science , 2018 .

[11]  Jeffrey M. Bradshaw,et al.  Tomorrow’s Human–Machine Design Tools: From Levels of Automation to Interdependencies , 2018 .

[12]  Alan Borning,et al.  A Survey of Value Sensitive Design Methods , 2018, Found. Trends Hum. Comput. Interact..

[13]  Dafna Shahaf,et al.  Analogy Mining for Specific Design Needs , 2017, CHI.

[14]  Gianluca Demartini,et al.  An Introduction to Hybrid Human-Machine Information Systems , 2017, Found. Trends Web Sci..

[15]  Aaron Halfaker,et al.  Operationalizing Conflict and Cooperation between Automated Software Agents in Wikipedia , 2017, Proc. ACM Hum. Comput. Interact..

[16]  Krzysztof Z. Gajos,et al.  Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation , 2017, Creativity & Cognition.

[17]  L. Winner DO ARTIFACTS HAVE (cid:1) POLITICS? , 2022 .

[18]  Erin Walker,et al.  The Effect of Peripheral Micro-tasks on Crowd Ideation , 2017, CHI.

[19]  Heiko Paulheim,et al.  Knowledge graph refinement: A survey of approaches and evaluation methods , 2016, Semantic Web.

[20]  Eric Horvitz,et al.  Uncertainty, Action, and Interaction: In Pursuit of Mixed-Initiative Computing , 2016 .

[21]  J. Grudin,et al.  Human-computer integration , 2016, Interactions.

[22]  Krzysztof Z. Gajos,et al.  IdeaHound: Improving Large-scale Collaborative Ideation with Crowd-Powered Real-time Semantic Modeling , 2016, UIST.

[23]  John Danaher,et al.  The Threat of Algocracy: Reality, Resistance and Accommodation , 2016, Philosophy & Technology.

[24]  Toby Walsh,et al.  Turing's red flag , 2015, Commun. ACM.

[25]  Donghee Yoo,et al.  An Ontology-based Co-creation Enhancing System for Idea Recommendation in an Online Community , 2015, DATB.

[26]  Krzysztof Z. Gajos,et al.  Providing Timely Examples Improves the Quantity and Quality of Generated Ideas , 2015, Creativity & Cognition.

[27]  Krzysztof Z. Gajos,et al.  Toward Collaborative Ideation at Scale: Leveraging Ideas from Others to Generate More Creative and Diverse Ideas , 2015, CSCW.

[28]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[29]  Judith S. Olson,et al.  Ways of Knowing in HCI , 2014, Springer New York.

[30]  Francesco Piazza,et al.  Pundit: augmenting web contents with semantics , 2013, Lit. Linguistic Comput..

[31]  Pablo N. Mendes,et al.  Improving efficiency and accuracy in multilingual entity extraction , 2013, I-SEMANTICS '13.

[32]  Jill Palzkill Woelfer,et al.  A value sensitive action-reflection model: evolving a co-design space with stakeholder and designer prompts , 2013, CHI.

[33]  Yann Mathet,et al.  The Glozz platform: a corpus annotation and mining tool , 2012, DocEng '12.

[34]  Eric Horvitz,et al.  Combining human and machine intelligence in large-scale crowdsourcing , 2012, AAMAS.

[35]  Aaron Halfaker,et al.  Bots and Cyborgs: Wikipedia's Immune System , 2012, Computer.

[36]  Li Chen,et al.  A user-centric evaluation framework for recommender systems , 2011, RecSys '11.

[37]  John Riedl,et al.  Introduction to the Transactions on Interactive Intelligent Systems , 2011, ACM Trans. Interact. Intell. Syst..

[38]  Christian Bizer,et al.  DBpedia spotlight: shedding light on the web of documents , 2011, I-Semantics '11.

[39]  Maarten Sierhuis,et al.  Beyond Cooperative Robotics: The Central Role of Interdependence in Coactive Design , 2011, IEEE Intelligent Systems.

[40]  R. Stuart Geiger,et al.  The work of sustaining order in wikipedia: the banning of a vandal , 2010, CSCW '10.

[41]  Matti Tedre,et al.  FEATUREWhat should be automated? , 2008, INTR.

[42]  Christopher D. Wickens,et al.  Humans: Still Vital After All These Years of Automation , 2008, Hum. Factors.

[43]  Viswanath Venkatesh,et al.  Technology Acceptance Model 3 and a Research Agenda on Interventions , 2008, Decis. Sci..

[44]  Sidney W. A. Dekker,et al.  MABA-MABA or Abracadabra? Progress on Human–Automation Co-ordination , 2002, Cognition, Technology & Work.

[45]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[46]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[47]  Ben Shneiderman,et al.  Direct manipulation vs. interface agents , 1997, INTR.

[48]  David B. Kaber,et al.  The Combined Effect of Level of Automation and Adaptive Automation on Human Performance with Complex, Dynamic Control Systems , 1997 .

[49]  Batya Friedman,et al.  Value-sensitive design , 1996, INTR.

[50]  Mica R. Endsley,et al.  The Out-of-the-Loop Performance Problem and Level of Control in Automation , 1995, Hum. Factors.

[51]  J. Johnson Mixing Humans and Nonhumans Together: The Sociology of a Door-Closer , 1988 .

[52]  Thomas B. Sheridan,et al.  Human and Computer Control of Undersea Teleoperators , 1978 .

[53]  Bruce W. Arden,et al.  The computer science and engineering research study (COSERS) , 1976, CACM.

[54]  N. Jordan Allocation of functions between man and machines in automated systems. , 1963 .

[55]  Douglas C. Engelbart,et al.  Augmenting human intellect: a conceptual framework , 1962 .

[56]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[57]  J. C. R. Licklider,et al.  Man-Computer Symbiosis , 1960 .

[58]  S S Stevens,et al.  HUMAN ENGINEERING FOR AN EFFECTIVE AIR-NAVIGATION AND TRAFFIC-CONTROL SYSTEM, AND APPENDIXES 1 THRU 3 , 1951 .

[59]  Ivan Lopez-Arevalo,et al.  Information extraction meets the Semantic Web: A survey , 2020, Semantic Web.

[60]  Claudia Müller-Birn,et al.  Innovonto: An Enhanced Crowd Ideation Platform with Semantic Annotation (Hallway Test) , 2018 .

[61]  Claudia Müller-Birn,et al.  Enabling Structured Data Generation by Nontechnical Experts , 2017, MuC.

[62]  O. Bjelland,et al.  An Inside View of IBM's 'Innovation Jam' , 2008 .

[63]  Charles E. Thorpe,et al.  Collaborative control: a robot-centric model for vehicle teleoperation , 2001 .

[64]  Charles L. Isbell,et al.  An IP Continuum for Adaptive Interface Design , 1996 .

[65]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[66]  Bruce W. Arden,et al.  What Can Be Automated?: Computer Science and Engineering Research Study , 1980 .