Evaluation of a Decentralized Architecture for Large Scale Collaborative Intrusion Detection

An important problem in network intrusion detection is how to detect large scale coordinated attacks such as scans, worms and denial-of-service attacks. These coordinated attacks can be difficult to detect at an early stage, since the evidence of the attack may be widely distributed across different subnetworks in the Internet. A critical issue for research is how to detect these large scale attacks by correlating information from multiple intrusion detection systems in an efficient manner. Several collaborative detection systems have been proposed in the literature. However, these proposals have lacked large scale testing in real networks, and the practicalities of how to optimize the trade-off between detection accuracy and reaction time of these systems has not been demonstrated. To address these challenges, we propose LarSID, a scalable decentralized large scale intrusion detection framework. LarSID provides a service for defending against attacks by sharing potential evidence of intrusions between participant intrusion detection systems via a distributed hash table (DHT) architecture. In particular, we investigate how to optimize the trade-off between detection accuracy and reaction time of LarSID based on an analysis of a large, real-world intrusion detection dataset (DShield Dataset), which has been collected from over 1600 firewall administrators across the world. LarSID has been deployed and tested on the PlanetLab testbed, and is built on top of OpenDHT - a public DHT service. Our experimental results show significant reductions in detection latency compared to a centralized detection architecture. Currently, LarSID has been deployed on 128 PlanetLab nodes as a large scale intrusion detection service.