Network-wide Information Correlation and Exploration ( NICE ) : Framework , Applications , and Experience

Scalable event detection and trouble shooting capabilities are critical for ensuring high levels of network reliability and performance. Although network operations systems are typically well designed for dealing with hard network outages (e.g., link failures), detecting and analyzing chronic conditions particularly those associated with short term performance impairments still remains challenging. Detecting and trouble shooting such conditions typically requires detailed analysis of data collected from different monitoring tools, to obtain a comprehensive view of network events. This is typically performed manually, making it an imperfect, time consuming and costly process. The ability to perform correlations is a fundamental yet powerful building block when it comes to analyzing multiple data series collectively. We present a novel framework, NICE (Network-wide Information Correlation and Exploration), that scalably analyzes network-wide statistical event correlations. The core components of NICE include a flexible infrastructure for pair-wise correlation testing as well as tools for subsequent analysis of resulting correlation patterns and automatic drill-down for surprising correlations. Above our core NICE infrastructure, we have prototyped two exciting applications: (i) for trouble-shooting known problems, and (ii) for discovering undesirable modes of network operation that may traditionally have been flying under the operations team’s radar, yet potentially impacting customers. We evaluate the accuracy of NICE by examining several data streams from a tier-1 ISP backbone network. We also present case studies that demonstrate the efficacy of our tool-kit by revealing surprising correlations for the same tier-1 network. The NICE methodology and algorithms promise to be of immense use to network operators in analyzing network behavior and identifying anomalous network conditions.