A heuristic correlation algorithm for data reduction through noise detection in stream-based communication management systems

Monitoring and management of modern telecommunication networks has become more and more challenging due to the explosion in scale of data generated by network elements. Not only has the size of the network, the number of nodes, and the number of customers increased, but the amount and dimensionality of the data coming from each managed element has also increased. To support sophisticated monitoring and management strategies it is desirable to forward as much trace data as possible into operators' operations support systems (Operations Support Systems (OSSs)). In this paper a heuristic algorithm is presented which reduces the data-stream by removing uncorrelated noise events by determining the degree of inter-relationship between the events in the data-stream. With a sophisticated open source control plane emulator used as the source generator, the results show that the presented algorithm is capable of differentiating noise from useful information thus significantly reducing scale and dimensionality of network monitoring data-streams.