Benchmarking Pub/Sub IoT middleware platforms for smart services

Middleware is being extensively used in Internet of Things (IoT) deployments and is available in a variety of flavors. Despite this extensive use and diversity, a fair comparison of the benefits, disadvantages, and performance of each middleware platform is missing. This comparison is relevant to support the decision process for IoT infrastructure. In this paper, we propose a set of qualitative and quantitative dimensions for benchmarking IoT middleware. We use the publication–subscription of a large dataset as use case inspired by a smart city scenario to compare two middleware platforms with standard ambition: FIWARE and oneM2M. We take these metrics and use case and systematically compare the two middleware platforms in the wild. We identify inefficiencies in implementations and characterize performance variations throughout the day, showing that the metrics may also be used for monitoring. Furthermore, we apply the same metrics and use case to two brokers set up in a controlled environment, providing infrastructure- and networking-independent insights. Finally, we summarize useful practical know-how acquired in the process that can speed up entrance into the topic and avoid configuration and implementation pitfalls that impact performance.