Wormhole: a novel big data platform for 100 Gbit/s network monitoring and beyond

Internet measurement and analysis is increasingly challenging as the Internet evolves, primarily due to changing-trends, speed increments or new protocols and ciphers. As such, ad-hoc monitoring equipment comes in handy, albeit cost-effectiveness impedes deployment at a very large scale. As an alternative, big data-based distributed architectures are being proposed for network monitoring and analysis. However, in light of the high throughput currently offered by 100 Gbit/s links, it turns out that state-of-the-art big data solutions fall short of capacity, unless a huge amount of computers are used. In order to effectively tackle that issue, we have created Wormhole: a streaming engine that circumvents existing limitations by distributing the input messages/packets coherently among different off-the-shelf analysis equipment, thus reducing costs and equipment. Should the incoming data rate be larger than the system throughput, a distributed file system can be used for temporary data storage, for subsequent filtering and in-depth analysis. The proposed solution provides on-line real-time monitoring metrics with the ability to gain further insights when required. The prototyped architecture is able to deal with 100 Gbit/s networks and can be easily scaled up to higher rates by just adding more computing nodes and/or by trimming encrypted packet payloads.