A novel reputation system to detect DGA-based botnets

A botnet is a network of compromised hosts (bots) remotely controlled by a so-called bot herder through one or more command and control (C&C) servers. New generation botnets, such as Conficker and Murofet, tend to use a form of domain fluxing for command and control. Each domain fluxing bot generates a list of domain names using a domain name generation algorithm (DGA) and queries each of them until one of them is resolved to a C&C server. Since the bot herder registers only a few of these domain names, the domain fluxing bots generate many failed DNS queries. Even though some efforts have been focused on the detection of DGA-based botnets, but none of them consider the history of suspicious activities. This makes the detection system has a potentially high false alarm rate. In this paper, we propose a novel reputation system to detect DGA-based botnets. Our main goal is to automatically assign a high negative reputation score to each host that is involved in suspicious bot activities. To achieve this goal, we first choose DNS queries with similar characteristics at the end of each time window. We then identify hosts that algorithmically generated a large set of suspicious domain names and add them to a so-called suspicious group activity matrix. We also identify hosts with high numbers of failed DNS queries and add them to a so called suspicious failure matrix. We finally calculate the negative reputation score of each host in these two matrices and detect hosts with high negative reputation scores as bot-infected. We evaluate our reputation system using DNS queries collected from the campus network. The experimental results show that it can successfully detect DGA-based botnets with a high detection rate and a low false alarm rate while providing real-time monitoring in large-scale networks.