Towards Early Detection of Novel Attack Patterns through the Lens of a Large-Scale Darknet

Darknet monitoring provides a cost-effective way to monitor the global trend of cyber-threats in the Internet. To make full use of the darknet traffic at hand, in this paper, we present a study on early detection of emerging novel attacks observed in the darknet. First, exploration of the regularities in the communications from attacking hosts are done by feeding all observed packets in the darknet to a frequent itemset mining engine, where the most frequently occurred attack patterns are automatically grouped together. Second, a time series which characterizes the activity level of each attack pattern is created over the observation period. Then, to extract the most prominent attack patterns, a clustering algorithm is engaged to cluster the attack patterns into groups that carry the similar activities in a long run, dimension reduction is employed to provide visual hints about their relationship. Finally, attacks featured by a recent rapid increase are picked up to be further inspected by security experts for incident handling purpose. The experiments show that the proposed scheme is effective, efficient in early detection of new attack patterns from conventional approaches.