Message Latency-Based Load Shedding Mechanism in Apache Kafka

Apache Kafka is a distributed message queuing platform that delivers data streams in real time. Through the distributed processing technology, Kafka has the advantage of delivering very large data streams very fast. However, when the data explosion occurs, the message latency largely increases and the system might be interrupted. This paper proposes a load shedding engine of Kafka that solves this message latency problem. The load shedding engine solves the data explosion problem by introducing a simple mechanism that restricts the transmission of some messages when the latency exceeds the given threshold in the Kafka’s producer. Experiments with Apache Storm-based real-time applications show that the latency does not continuously increase due to the load shedding function in both single and multiple data streams, and maintains a constant level. This is the first attempt to apply a load shedding technique to Kafka-based real-time stream processing, providing simple and efficient data explosion control.