Enhanced telemetry for encrypted threat analytics

Traditional flow monitoring provides a high-level view of network communications by reporting the addresses, ports, and byte and packet counts of a flow. This data is valuable, but it gives little insight into the actual content or context of a flow. To obtain this missing insight, we investigated intra-flow data, that is, information about events that occur inside of a flow that can be conveniently collected, stored, and analyzed within a flow monitoring framework. The focus of our work is on new types of data that are independent of protocol details, such as the lengths and arrival times of messages within a flow. These data elements have the attractive property that they apply equally well to both encrypted and unencrypted flows. Protocol-aware telemetry, specifically TLS-aware telemetry, is also analyzed. In this paper, we explore the benefits of enhanced telemetry, desirable properties of new intra-flow data features with respect to a flow monitoring system, and how best to use machine learning classifiers that operate on this data. We provide results on millions of flows processed by our open source program. Finally, we show that leveraging appropriate data features and simple machine learning models can successfully identify threats in encrypted network traffic.