Adaptive Crawling with Multiple Bots: A Matroid Intersection Approach

In this work, we examine the problem of adaptively uncovering network topology in an incomplete network, to support more accurate decision making in various real-world applications, such as modeling for reconnaissance attacks and network probing. While this problem has been partially studied, we provide a novel take on it by modeling it with a set of crawlers termed “bots” which can uncover independent portions of the network in parallel. Accordingly, we develop three adaptive algorithms, which make decisions based on previous observations due to incomplete information, namely AGP, a sequential method; FastAGP, a parallel algorithm; and ALSP, an extension of FastAGP uses local search to improve guarantees. These algorithms are proven to have 1/3, 1/7, and 1/ (5 + ∊) approximation ratios, respectively. The key analysis of these algorithms is the connection between adaptive algorithms and an intersection of multiple partition matroids. We conclude with an evaluation of these algorithms to quantify the impact of both adaptivity and parallelism. We find that in practice, adaptive approaches perform significantly better, while FastAGP performs nearly as well as AGP in most cases despite operating in a massively parallel fashion. Finally, we show that a balance between the quantity and quality of bots is ideal for maximizing observation of the network.