A fast and accurate threat detection and prevention architecture using stream processing

Late detection of security breaches increases the risk of irreparable damages and limits any mitigation attempts. We propose a fast and accurate threat detection and prevention architecture that combines the advantages of real‐time streaming with batch processing over a historical database. We create a dataset by capturing both legitimate and malicious traffic and propose two ways of combining packets into flows, one considering a time window and the other analyzing the first few packets of each flow per period. We also investigate the effectiveness of our proposal on real‐world network traces obtained from a significant Brazilian network operator providing broadband Internet to their customers. We implement and evaluate three classification algorithms and two anomaly detection methods. The results show an accuracy higher than 95% and an excellent trade‐off between attack detection and false‐positive rates. We further propose an improved scheme based on software defined networks that automatically prevents threats by analyzing only the first few packets of a flow. The proposal promptly and efficiently blocks threats, is robust, and can scale up, even when the attacker employs spoofed IP.