Behavioral Recognition and Prediction of an Operator Supervising Multiple Heterogeneous Unmanned Vehicles

The ability to recognize patterns of operator behavior that could lead to poor outcomes is critical to monitoring the overall performance of the human-unmanned system team. We propose a method that relies on Bayesian machine learning in order to automatically derive a set of states that describe the behavior of an operator. More specifically, we use the Hidden Markov Model (HMM) formalism to infer higher cognitive states from observable operator interaction with a computer interface. This allows the categorization of a pattern of action over a probability distribution of possible operator states which can then be correlated with a range of mission outcomes. Moreover, the HMM provides the means to project in the future and therefore aid in prediction, in probabilistic terms, of the future actions of an operator. In this paper, we present the methodology used to derive the operator model and show initial results based on experimental data.