The Responsibility Quantification Model of Human Interaction With Automation

Intelligent systems and advanced automation are involved in information collection and evaluation, decision-making, and the implementation of chosen actions. In such systems, human responsibility becomes equivocal. Understanding human causal responsibility is particularly important when systems can harm people, as with autonomous vehicles or, most notably, with autonomous weapon systems (AWSs). Using information theory, we developed a responsibility quantification (ResQu) model of human causal responsibility in intelligent systems and demonstrated its applications on decisions regarding AWS. The analysis reveals that comparative human responsibility for outcomes is often low, even when major functions are allocated to the human. Thus, broadly stated policies of keeping humans in the loop and having meaningful human control are misleading and cannot truly direct decisions on how to involve humans in advanced automation. The current model assumes stationarity, full knowledge regarding the characteristic of the human and automation, and ignores temporal aspects. It is an initial step toward the development of a comprehensive responsibility model that will make it possible to quantify human causal responsibility. The model can serve as an additional tool in the analysis of system design alternatives and policy decisions regarding human causal responsibility, providing a novel, quantitative perspective on these matters. Note to Practitioners—We developed a theoretical model and a quantitative measure for computing the comparative human causal responsibility in the interaction with intelligent systems and advanced automation. Our responsibility measure can be applied by practitioners (system designers, regulators, and so on) to estimate user responsibility in specific system configurations. This can serve as an additional tool in the comparison between alternative system designs or deployment policies, by relating different automation design options to their predicted effect on the users’ responsibility. To apply the model (which is based on entropy and mutual information) to real-world systems, one must deduce the underlying distributions, either from known system properties or from empirical observations, taken over time. The initial version of the model we present here assumes that the combined human–automation system is stationary and ergodic. Real-world systems may not be stationary and ergodic or cannot be observed sufficiently to allow accurate estimates of the required input of multivariate probabilities, in which case the computed responsibility values should be treated with caution. Nevertheless, the construction of a ResQu information flow model, combined with sensitivity analyses of how changes in the input probabilities and assumptions affect the responsibility measure, will often reveal important qualitative properties and supply valuable insights regarding the general level of meaningful human involvement and comparative responsibility in a system.