Practical Algorithms for Size Estimation in Large and Dynamic Groups

Large-scale distributed systems may be required to estimate the number of non-faulty processes present in the group at a given point of time. The problem is related to aggregation, and several solutions have been proposed. In this paper, we present two new sampling-based algorithms for estimation in dynamic groups (i.e., where processes are constantly joining and crashing), and thoroughly evaluate them using real-life traces. One scheme spreads a gossip into the overlay first, and then samples the receipt times of this gossip at different processes. The second scheme measures the density of processes when their identifiers are hashed into a real interval. The schemes have low latency, per-process overheads, while providing high levels of probabilistic accuracy. We present simulation studies that measure and compare the performance of these approaches in static groups, and groups with significant turnaround (arrival and departure) of processes. The latter are done by using traces from deployed peer to peer overlays. The schemes are generic enough to be used by any distributed application. Student Paper : Yes (Dimitrios Psaltoulis and Dionysios Kostoulas are students)