Batch Inverse Reinforcement Learning Using Counterfactuals for Understanding Decision Making

A key challenge in modeling real-world decision-making is the fact that active experimentation is often impossible (e.g. in healthcare). The goal of batch inverse reinforcement learning is to recover and understand policies on the basis of demonstrated behaviour--i.e. trajectories of observations and actions made by an expert maximizing some unknown reward function. We propose incorporating counterfactual reasoning into modeling decision behaviours in this setting. At each decision point, counterfactuals answer the question: Given the current history of observations, what would happen if we took a particular action? First, this offers a principled approach to learning inherently interpretable reward functions, which enables understanding the cost-benefit tradeoffs associated with an expert's actions. Second, by estimating the effects of different actions, counterfactuals readily tackle the off-policy nature of policy evaluation in the batch setting. Not only does this alleviate the cold-start problem typical of conventional solutions, but also accommodates settings where the expert policies are depending on histories of observations rather than just current states. Through experiments in both real and simulated medical environments, we illustrate the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of expert behaviour.