Modeling Human-AI Team Decision Making

AI and humans bring complementary skills to group deliberations. Modeling this group decision making is especially challenging when the deliberations include an element of risk and an exploration-exploitation process of appraising the capabilities of the human and AI agents. To investigate this question, we presented a sequence of intellective issues to a set of human groups aided by imperfect AI agents. A group’s goal was to appraise the relative expertise of the group’s members and its available AI agents, evaluate the risks associated with different actions, and maximize the overall reward by reaching consensus. We propose and empirically validate models of human-AI team decision making under such uncertain circumstances, and show the value of sociocognitive constructs of prospect theory, influence dynamics, and Bayesian learning in predicting the behavior of humanAI groups.

[1]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[2]  Jens Rasmussen,et al.  Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[3]  A. Tversky,et al.  Advances in prospect theory: Cumulative representation of uncertainty , 1992 .

[4]  Thomas B. Sheridan,et al.  Human and Computer Control of Undersea Teleoperators , 1978 .

[5]  A. W. Evans,et al.  Bidirectional Communication for Effective Human-Agent Teaming , 2018, HCI.

[6]  M R Endsley,et al.  Level of automation effects on performance, situation awareness and workload in a dynamic control task. , 1999, Ergonomics.

[7]  Yashesh Gaur,et al.  The effects of automatic speech recognition quality on human transcription latency , 2016, W4A.

[8]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[9]  H. Francis Song,et al.  Machine Theory of Mind , 2018, ICML.

[10]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[11]  Francesco Bullo,et al.  Mathematical Structures in Group Decision-Making on Resource Allocation Distributions , 2019, Scientific Reports.

[12]  E. W. Adams,et al.  Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting , 1962 .

[13]  Danielle S Bassett,et al.  Detection of functional brain network reconfiguration during task-driven cognitive states , 2016, NeuroImage.

[14]  J. Townsend,et al.  The Oxford Handbook of Computational and Mathematical Psychology , 2015 .

[15]  Beau G. Schelble,et al.  Human–Autonomy Teaming: A Review and Analysis of the Empirical Literature , 2020, Hum. Factors.

[16]  Kaleb McDowell,et al.  Enhancing HumanAgent Teaming with Individualized, Adaptive Technologies: A Discussion of Critical Scientific Questions , 2018 .

[17]  D. Wegner.  Transactive Memory: A Contemporary Analysis of the Group Mind , 1987 .

[18]  Jean M. Vettel,et al.  Estimating direction in brain-behavior interactions: Proactive and reactive brain states in driving , 2016, NeuroImage.

[19]  Dayong Wang,et al.  Deep Learning for Identifying Metastatic Breast Cancer , 2016, ArXiv.

[20]  David Haussler,et al.  How to use expert advice , 1993, STOC.

[21]  Mary Missy Cummings,et al.  Man versus Machine or Man + Machine? , 2014, IEEE Intelligent Systems.

[22]  Jessie Y. C. Chen,et al.  Human–Agent Teaming for Multirobot Control: A Review of Human Factors Issues , 2014, IEEE Transactions on Human-Machine Systems.

[23]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[24]  Adam See,et al.  Does the Chimpanzee Have Theory of Mind? , 2021, Encyclopedia of Evolutionary Psychological Science.

[25]  Eric Horvitz,et al.  Combining human and machine intelligence in large-scale crowdsourcing , 2012, AAMAS.

[26]  J. Townsend,et al.  Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment. , 1993, Psychological review.

[27]  Nicholas A. Christakis,et al.  Locally noisy autonomous agents improve global human coordination in network experiments , 2017, Nature.

[28]  Gideon Keren,et al.  Perspectives on thinking, judging, and decision making , 2011 .

[29]  Iyad Rahwan,et al.  Cooperating with machines , 2017, Nature Communications.

[30]  M. Degroot.  Reaching a Consensus , 1974 .

[31]  Tim Miller,et al.  Communication in Human-Agent Teams for Tasks with Joint Action , 2015, COIN@AAMAS/IJCAI.

[32]  B. Sahakian,et al.  Knowing me, knowing you: theory of mind in AI , 2020, Psychological Medicine.

[33]  J. R. French,et al.  The bases of social power. , 1959 .

[34]  Joshua B. Tenenbaum,et al.  Bayesian Theory of Mind: Modeling Joint Belief-Desire Attribution , 2011, CogSci.

[35]  Francesco Bullo,et al.  Dynamic models of appraisal networks explaining collective learning , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[36]  Kaleb McDowell,et al.  The privileged sensing framework: A principled approach to improved human-autonomy integration , 2018 .

[37]  GUlden Ulkiimen,et al.  DISTINGUISHING TWO DIMENSIONS OF UNCERTAINTY , 2011 .

[38]  Noah E. Friedkin,et al.  Social Influence Network Theory: A Sociological Examination of Small Group Dynamics , 2011 .

[39]  Pamela J. Hinds,et al.  Autonomy and Common Ground in Human-Robot Interaction: A Field Study , 2007, IEEE Intelligent Systems.

[40]  H. Simon,et al.  Models Of Man : Social And Rational , 1957 .

[41]  David Engel,et al.  What Makes a Strong Team?: Using Collective Intelligence to Predict Team Performance in League of Legends , 2017, CSCW.

[42]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[43]  Eric Horvitz,et al.  Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance , 2019, HCOMP.

[44]  D. Meyer,et al.  Supporting Online Material Materials and Methods Som Text Figs. S1 to S6 References Evidence for a Collective Intelligence Factor in the Performance of Human Groups , 2022 .

[45]  E. Wagenmakers,et al.  Hierarchical Bayesian parameter estimation for cumulative prospect theory , 2011, Journal of Mathematical Psychology.

[46]  R. M. Adelson,et al.  Utility Theory for Decision Making , 1971 .

[47]  S. Chaiken,et al.  Dual-process theories in social psychology , 1999 .

[48]  Katia Sycara,et al.  Deep Interpretable Models of Theory of Mind For Human-Agent Teaming , 2021, ArXiv.

[49]  Adele Diederich,et al.  Models of Decision Making under Risk and Uncertainty , 2015 .

[50]  A. Kiureghian,et al.  Aleatory or epistemic? Does it matter? , 2009 .