Micro-signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection

Network intrusion detection is broadly divided into signature and anomaly detection. The former identifies patterns associated with known attacks and the latter attempts to learn a ‘normal’ pattern of activity and alerts when behaviors outside of those norms is detected. The n-gram methodology has arguably been the most successful technique for network anomaly detection. In this work we discover that when training data is sanitized, n-gram anomaly detection is not primarily anomaly detection, as it receives the majority of its performance from an implicit non-anomaly subsystem, that neither uses typical signatures nor is anomaly based (though it is closely related to both). We find that for our data, these “micro-signatures” provide the vast majority of the detection capability. This finding changes how we understand and approach n-gram based ‘anomaly’ detection. By understanding the foundational principles upon which it operates, we can then better explore how to optimally improve it.