Clear and Present Data: Opaque Traffic and its Security Implications for the Future

Opaque traffic, i.e., traffic that is compressed or encrypted, incurs particularly high overhead for deep packet inspection engines and often yields little or no useful information. Our experiments indicate that an astonishing 89% of payload-carrying TCP packets — and 86% of bytes transmitted — are opaque, forcing us to consider the challenges this class of traffic presents for network security, both in the short-term and, as the proportion of opaque traffic continues to rise, for the future. We provide a first step toward addressing some of these challenges by introducing new techniques for accurate real-time winnowing, or filtering, of such traffic based on the intuition that the distribution of byte values found in opaque traffic will differ greatly from that found in transparent traffic. Evaluation on traffic from two campuses reveals that our techniques are able to identify opaque data with 95% accuracy, on average, while examining less than 16 bytes of payload data. We implemented our most promising technique as a preprocessor for the Snort IDS and compared the performance to a stock Snort instance by running both instances live, on identical traffic streams, using a Data Acquisition and Generation (DAG) card deployed within a campus network. Winnowing enabled Snort to handle a peak load of 1.2Gbps, with zero percent packet loss, and process almost one hundred billion packets over 24 hours — a 147% increase over the number processed by the stock Snort instance. This increase in capacity resulted in 33,000 additional alerts which would otherwise have been missed.