A Clinical Trial Derived Reference Set for Evaluating Observational Study Methods

Informing clinical practice in a learning health system requires good causal inference methods in order to estimate the effects of drugs from observational studies. However, these methods rely on untestable assumptions, making it difficult to evaluate whether the returned drug effect estimates from a method are reliable, or to compare the effectiveness of different methods. In this work, we provide a reference set of the effects of drugs on various side effects based on public clinical trials data. We improve on prior reference sets by constructing a consistent statistical definition of positive and negative controls and by constructing our controls from clinical trials instead of drug package inserts or the literature. We also provide an example application, where we use the reference set to evaluate a suite of causal inference methods on observational medical claims data. In doing so, we find that the treatment effects estimated using inverse propensity weighting with propensity scores estimated via machine learning accurately separate the positive controls from the negative controls in our reference set.