Inferring the Security Performance of Providers from Noisy and Heterogenous Abuse Datasets

Abuse data offers one of the very few empirical measurements of the security performance of defenders. As such, it can play an important role in strengthening and aligning the security incentives in a variety of markets. Using abuse data to measure security performance suffers from a number of problems, however. Abuse data is notoriously noisy, highly heterogeneous, often incomplete, biased, and driven by a multitude of causal factors that are hard to disentangle. We present the first comprehensive approach to measure defender security performance from a combination of heterogeneous abuse datasets, taking all of these issues into account. We present a causal model of incidents, test for biases across seven abuse datasets and then propose a new modeling approach. Using Item Response Theory, we estimate the security performance of providers as a latent, unobservable trait. The approach also allows us to quantify the uncertainty of the performance estimates. Despite the uncertainties, we demonstrate the effectiveness of the approach by using the security performance estimates to predict a large portion of the variance in the abuse counts observed in independent datasets, after controlling for various exposure effects such as the size and business type of the providers.