Trusted Partners: Human-Machine Teaming and the Future of Military AI

As the U.S. military integrates artificial intelligence into its systems and missions, there are outstanding questions about the role of trust in human-machine teams. This report examines the drivers and effects of such trust, assesses the risks from too much or too little trust in intelligent technologies, reviews efforts to build trustworthy AI systems, and offers future directions for research on trust relevant to the U.S. military.

[1]  Berkeley J. Dietvorst,et al.  Algorithm Aversion: People Erroneously Avoid Algorithms after Seeing Them Err , 2014, Journal of experimental psychology. General.

[2]  John D. Lee,et al.  Cooperation in Human-Agent Systems to Support Resilience , 2016, Hum. Factors.

[3]  Guy Hoffman,et al.  Machines as a source of consolation: Robot responsiveness increases human approach behavior and desire for companionship , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[4]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[5]  Robin Williams,et al.  Trust in Technology , 2006 .

[6]  Logan Engstrom,et al.  Black-box Adversarial Attacks with Limited Queries and Information , 2018, ICML.

[7]  Linda Onnasch,et al.  Human Performance Consequences of Automated Decision Aids , 2012 .

[8]  Owen Brown,et al.  Defense Advanced Research Projects Agency , 2009 .

[9]  Kathleen L. Mosier,et al.  Does automation bias decision-making? , 1999, Int. J. Hum. Comput. Stud..

[10]  J. Simmons,et al.  Algorithm Aversion , 2016 .

[11]  Keng Siau,et al.  Building Trust in Artificial Intelligence, Machine Learning, and Robotics , 2018 .

[12]  Mark A. Neerincx,et al.  Trust Development and Repair in Human-Robot Teams , 2017, HRI.

[13]  Lynne E. Parker,et al.  Creation of the National Artificial Intelligence Research and Development Strategic Plan , 2018, AI Mag..

[14]  Dr. John K. Hawley.  PATRIOT WARS Automation and the Patriot Air and Missile Defense System , 2017 .

[15]  Neera Jain,et al.  Real-Time Sensing of Trust in Human-Machine Interactions , 2016 .

[16]  Mary L. Cummings,et al.  Automation Bias in Intelligent Time Critical Decision Support Systems , 2004 .

[17]  A. W. Evans,et al.  Communicating intent to develop shared situation awareness and engender trust in human-agent teams , 2017, Cognitive Systems Research.

[18]  David Danks,et al.  “Trust but Verify”: The Difficulty of Trusting Autonomous Weapons Systems , 2018 .

[19]  Francesca Rossi,et al.  Building Trust in Artificial Intelligence , 2018 .

[20]  Holly A. Yanco,et al.  Impact of robot failures and feedback on real-time trust , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[21]  Nadine B. Sarter,et al.  Supporting Trust Calibration and the Effective Use of Decision Aids by Presenting Dynamic System Confidence Information , 2006, Hum. Factors.

[22]  Michael A. Rupp,et al.  Intelligent Agent Transparency in Human–Agent Teaming for Multi-UxV Management , 2016, Hum. Factors.

[23]  Bing Cai Kok,et al.  Trust in Robots: Challenges and Opportunities , 2020, Current Robotics Reports.

[24]  Raja Parasuraman,et al.  Complacency and Bias in Human Use of Automation: An Attentional Integration , 2010, Hum. Factors.

[25]  Gari Palmer,et al.  The “Trust V”: Building and Measuring Trust in Autonomous Systems , 2016 .

[26]  R. Smith,et al.  Department of Defense. , 2020, Military medicine.

[27]  Tatsuya Nomura,et al.  The influence of people’s culture and prior experiences with Aibo on their attitude towards robots , 2006, AI & SOCIETY.

[28]  Matthew W. Lewis,et al.  Automating Army Convoys, Technical and Tactical Risks and Opportunities , 2020 .

[29]  Nancy J. Cooke,et al.  Understanding the Role of Trust in Human-Autonomy Teaming , 2019, HICSS.

[30]  Claire La Fleur,et al.  Analysis of Facial Expressions: Explaining Affective State and Trust-Based Decisions during Interaction with Automation , 2020 .

[31]  Masooda N. Bashir,et al.  Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust , 2015, Hum. Factors.

[32]  Eric Jones,et al.  A Heuristic Based Framework for Improving Design of Unmanned Systems by Quantifying and Assessing Operator Trust , 2016 .

[33]  Jessie Y. C. Chen,et al.  Effects of Agent Transparency on human-autonomy teaming effectiveness , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[34]  Abdul V. Roudsari,et al.  Automation bias: a systematic review of frequency, effect mediators, and mitigators , 2012, J. Am. Medical Informatics Assoc..

[35]  William M. Campbell,et al.  Recommender Systems for the Department of Defense and Intelligence Community , 2016 .

[36]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004, Hum. Factors.

[37]  Francis Sullivan,et al.  Trust but verify , 2002, Computing in Science & Engineering.

[38]  Jessie Y. C. Chen,et al.  Agent Transparency and Reliability in Human–Robot Interaction: The Influence on User Confidence and Perceived Reliability , 2020, IEEE Transactions on Human-Machine Systems.

[39]  Michael W. Boyce,et al.  Situation Awareness-Based Agent Transparency , 2014 .

[40]  Anita Williams Woolley,et al.  Human Trust in Artificial Intelligence: Review of Empirical Research , 2020, Academy of Management Annals.

[41]  Yochanan E. Bigman,et al.  People are averse to machines making moral decisions , 2018, Cognition.

[42]  Ehsan Toreini,et al.  The relationship between trust in AI and trustworthy machine learning technologies , 2019, FAT*.

[43]  Chuck Howell,et al.  Key Considerations for the Responsible Development and Fielding of Artificial Intelligence , 2021, ArXiv.

[44]  Andrew J. Lohn,et al.  Military Applications of Artificial Intelligence: Ethical Concerns in an Uncertain World , 2020 .

[45]  Elizabeth Ann Severns,et al.  Office, Secretary of Defense , 2019, U.S. Defense Planning.

[46]  Ning Wang,et al.  Trust calibration within a human-robot team: Comparing automatically generated explanations , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[47]  Allies and Artificial Intelligence: Obstacles to Operations and Decision-Making (Spring 2020) , 2020 .

[48]  Deanne N. Den Hartog,et al.  Measuring trust inside organisations , 2006 .