Abnormal Hosts Monitor for City Wide Core Network by Real Time Super Points Cardinality Estimation

Core network abnormal events, such as warm, spam, scanning, ddos and so on, threat the security of the network. These abnormal events relate to a special kind of hosts named super points. For a host in the network, its cardinality is the number of other distinct hosts communicating with it during a certain time window. Super points, whose cardinalities are bigger than a predefined threshold, are a small part of hosts but play important roles for network management and security. This paper devises a novel paradigm to detect abnormal network events by monitoring super points. On the basis that normal events contain bi-direction packets, super points detected in different directions should be the same. So by comparing the difference of super points detected in different directions, abnormal events would be found out. The key step in the paradigm is to detect out super points in real time because super points in different directions need to be identified parallel. This paper proposes a double direction hash functions group which can map hosts randomly and restore them from a dense structure. Because the high randomness and simple process of the double direction hash functions group, this novel algorithm reduces the memory greatly, smaller than one-fourth of other algorithms' memory. This algorithm is also parallel available which means it can run on GPU to deal with huge packets in real time. Experiments on the traffic collecting from a city-wide network demonstrate the advantage of our algorithm.