Certifiable Trust in Autonomous Systems: Making the Intractable Tangible

This article discusses verification and validation (V&V) of autonomous systems, a concept that will prove to be difficult for systems that were designed to execute decision initiative. V&V of such systems should include evaluations of the trustworthiness of the system based on transparency inputs and scenario-based training. Transparency facets should be used to establish shared awareness and shared intent between the designer, tester, and user of the system. The transparency facets will allow the human to understand the goals, social intent, contextual awareness, task limitations, analytical underpinnings, and team-based orientation of the system in an attempt to verify its trustworthiness. Scenario-based training can then be used to validate that programming in a variety of situations that test the behavioral repertoire of the system. This novel method should be used to analyze behavioral adherence to a set of governing principles coded into the system.

[1]  Clint A. Bowers,et al.  The Impact of Cross-Training and Workload on Team Functioning: A Replication and Extension of Initial Findings , 1998, Hum. Factors.

[2]  Chieh-Wen Sheng,et al.  RELATIONSHIPS AMONG TEAMWORK BEHAVIOR, TRUST, PERCEIVED TEAM SUPPORT, AND TEAM COMMITMENT , 2010 .

[3]  Marvin Minsky,et al.  A framework for representing knowledge , 1974 .

[4]  Regina A. Pomranky,et al.  The role of trust in automation reliance , 2003, Int. J. Hum. Comput. Stud..

[5]  Jessica L. Wildman,et al.  Using Simulation-Based Training to Enhance Management Education , 2009 .

[6]  Anand K. Gramopadhye,et al.  Measurement of trust in complex and dynamic systems using a quantitative approach , 2004 .

[7]  F. Lateef.  Simulation-based learning: Just like the real thing , 2010, Journal of emergencies, trauma, and shock.

[8]  Christopher A. Miller,et al.  Trust and etiquette in high-criticality automated systems , 2004, CACM.

[9]  Aaron Powers,et al.  Matching robot appearance and behavior to tasks to improve human-robot cooperation , 2003, The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. ROMAN 2003..

[10]  Richard Pak,et al.  Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults , 2012, Ergonomics.

[11]  Alan R. Wagner,et al.  Timing Is Key for Robot Trust Repair , 2015, ICSR.

[12]  Stefan Rüping,et al.  Learning interpretable models , 2006 .

[13]  Chris Wilkinson,et al.  Final Report - Regulatory Considerations for Adaptive Systems. , 2013 .

[14]  J. H. Davis,et al.  An Integrative Model Of Organizational Trust , 1995 .

[15]  E. Salas,et al.  Toward an understanding of team performance and training. , 1992 .

[16]  P. Gollwitzer,et al.  Implementation intentions and goal achievement: A meta-analysis of effects and processes , 2006 .

[17]  Tae Young Han,et al.  Multilevel Investigation of Adaptive Performance , 2008 .

[18]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004 .

[19]  Pamela J. Hinds,et al.  Who Should I Blame? Effects of Autonomy and Transparency on Attributions in Human-Robot Interaction , 2006, ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication.

[20]  Gilad Chen,et al.  A multilevel examination of the relationships among training outcomes, mediating regulatory processes, and adaptive performance. , 2005, The Journal of applied psychology.

[21]  Paul Nielsen,et al.  Defense Science Board Summer Study on Autonomy , 2016 .

[22]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[23]  Michael A. Rupp,et al.  Intelligent Agent Transparency in Human–Agent Teaming for Multi-UxV Management , 2016, Hum. Factors.

[24]  Barry A. Bodt,et al.  Autonomy Community of Interest (COI) Test and Evaluation, Verification and Validation (TEVV) Working Group: Technology Investment Strategy 2015-2018 , 2015 .

[25]  Andrew Garland,et al.  Learning Hierarchical Task Models By Demonstration , 2002 .

[26]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[27]  Joseph Y. Halpern,et al.  Causes and Explanations: A Structural-Model Approach. Part I: Causes , 2000, The British Journal for the Philosophy of Science.

[28]  Andreas Holzinger,et al.  Learning performance with interactive simulations in medical education: Lessons learned from results of learning complex physiological models with the HAEMOdynamics SIMulator , 2009, Comput. Educ..

[29]  Joseph B. Lyons,et al.  Being Transparent about Transparency: A Model for Human-Robot Interaction , 2013, AAAI Spring Symposium: Trust and Autonomous Systems.

[30]  C. Burke,et al.  The impact of cross-training on team effectiveness. , 2002, The Journal of applied psychology.

[31]  Holly A. Yanco,et al.  Creating Trustworthy Robots: Lessons and Inspirations from Automated Systems , 2009 .

[32]  Joseph B. Lyons,et al.  Influence of Cultural, Organizational, and Automation Capability on Human Automation Trust: A Case Study of Auto-GCAS Experimental Test Pilots , 2014 .

[33]  David E. Smith,et al.  Engineering Trust in Complex Automated Systems , 2016 .

[34]  Stephen M Fiore,et al.  Augmenting team cognition in human-automation teams performing in complex operational environments. , 2007, Aviation, space, and environmental medicine.

[35]  Joseph Y. Halpern,et al.  Causes and explanations: A structural-model approach , 2000 .

[36]  Jessie Y. C. Chen,et al.  Agent Transparency for Human-Agent Teaming Effectiveness , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[37]  Huiyang Li,et al.  Human Performance Consequences of Stages and Levels of Automation , 2014, Hum. Factors.

[38]  Ronald C. Arkin,et al.  Governing Lethal Behavior in Autonomous Robots , 2009 .

[39]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[40]  Peter A. Hancock,et al.  Can You Trust Your Robot? , 2011 .

[41]  Ronald C. Arkin,et al.  Moral Decision Making in Autonomous Systems: Enforcement, Moral Emotions, Dignity, Trust, and Deception , 2012, Proceedings of the IEEE.

[42]  Michael Pressley,et al.  Scaffolding student learning: Instructional approaches and issues. , 1997 .

[43]  Lu Wang,et al.  Trust and Reliance on an Automated Combat Identification System , 2009, Hum. Factors.

[44]  Kerstin Fischer,et al.  How People Talk with Robots: Designing Dialog to Reduce User Uncertainty , 2011, AI Mag..

[45]  S. Puntambekar,et al.  Tools for Scaffolding Students in a Complex Learning Environment: What Have We Gained and What Have We Missed? , 2005 .

[46]  E. Salas,et al.  Cross-training and team performance. , 1998 .

[47]  Eric E. Geiselman,et al.  Flight Deck Automation , 2013 .

[48]  Pamela J. Hinds,et al.  Autonomy and Common Ground in Human-Robot Interaction: A Field Study , 2007, IEEE Intelligent Systems.

[49]  Christian Plaunt,et al.  An Emergency Landing Planner for Damaged Aircraft , 2009, IAAI.

[50]  Catherine E. Volpe,et al.  Defining Competencies and Establishing Team Training Requirements , 1995 .

[51]  Jessie Y. C. Chen,et al.  A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction , 2011, Hum. Factors.

[52]  Jessie Y. C. Chen,et al.  Supervisory Control of Multiple Robots: Human-Performance Issues and User-Interface Design , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[53]  Eduardo Salas,et al.  Does Team Training Improve Team Performance? A Meta-Analysis , 2008, Hum. Factors.

[54]  Nancy J. Cooke,et al.  Training Adaptive Teams , 2010, Hum. Factors.

[55]  Roy D. Pea,et al.  The Social and Technological Dimensions of Scaffolding and Related Theoretical Concepts for Learning, Education, and Human Activity , 2004, The Journal of the Learning Sciences.

[56]  E. Salas,et al.  Team Performance and Training in Complex Environments , 1998 .

[57]  Paul E. Spector,et al.  The Impact of Cross-Training on Team Functioning: An Empirical Investigation , 1996, Hum. Factors.

[58]  Kevin C. Stagl,et al.  Understanding team adaptation: a conceptual analysis and model. , 2006, The Journal of applied psychology.

[59]  Hector Muñoz-Avila,et al.  Hierarchical Plan Representations for Encoding Strategic Game AI , 2005, AIIDE.

[60]  Ericka Rovira,et al.  Understanding reliance on automation: effects of error type, error distribution, age and experience , 2014, Theoretical issues in ergonomics science.