Moving Beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems

Performance comparisons of low-power wireless systems are often not substantiated by accurate and realistic evaluations, which raises the need of a proper benchmark. In a first attempt towards a rigorous comparison of protocol performance under the exact same settings, we have developed in 2016 a prototype benchmarking infrastructure called D-Cube, and used it to run the first of a series of competitions aiming to quantitatively assess the performance of low-power wireless protocols in specific scenarios. Given the success of the competition among both academia and industry, we have significantly extended the benchmarking infrastructure in the following two editions: D-Cube now also supports, among others, remote experimentation, multiple traffic patterns and loads, a custom description of how to derive performance metrics, and is further able to control the network density as well as the harshness of the RF environment. In this paper we perform a critical analysis of the current capabilities of D-Cube and argue that its main limiting factor is that the traffic patterns and node identities are manually embedded in the source code by developers and cannot be changed automatically. We show that we can overcome this limitation by utilizing a well-known data structure and by having developers describe its memory address using a configuration file that is passed to the benchmarking infrastructure. Following this concept, we extend D-Cube with the ability of building and applying patches to binary files and show that this allows not only to automatically change traffic patterns and node identities, but to also change user-defined protocol parameters. We believe that this extension is one of the last missing stepping stones to make D-Cube a full-fledged benchmarking infrastructure for low-power wireless systems.