Efficient and reliable network tomography in heterogeneous networks using BitTorrent broadcasts and clustering algorithms

In the area of network performance and discovery, network tomography focuses on reconstructing network properties using only end-to-end measurements at the application layer. One challenging problem in network tomography is reconstructing available bandwidth along all links during multiple source/multiple destination transmissions. The traditional measurement procedures used for bandwidth tomography are extremely time consuming. We propose a novel solution to this problem. Our method counts the fragments exchanged during a BitTorrent broadcast. While this measurement has a high level of randomness, it can be obtained very efficiently, and aggregated into a reliable metric. This data is then analyzed with state-of-the-art algorithms, which correctly reconstruct logical clusters of nodes interconnected by high bandwidth, as well as bottlenecks between these logical clusters. Our experiments demonstrate that the proposed two-phase approach efficiently solves the presented problem for a number of settings on a complex grid infrastructure.

[1]  Sayantan Sur,et al.  Design and Evaluation of Network Topology-/Speed- Aware Broadcast Algorithms for InfiniBand Clusters , 2011, 2011 IEEE International Conference on Cluster Computing.

[2]  Jogesh K. Muppala,et al.  A Survey of BitTorrent Performance , 2010, IEEE Communications Surveys & Tutorials.

[3]  Songqing Chen,et al.  TopBT: A Topology-Aware and Infrastructure-Independent BitTorrent Client , 2010, 2010 Proceedings IEEE INFOCOM.

[4]  Benjamin H. Good,et al.  Performance of modularity maximization in practical contexts. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[5]  Thomas Hérault,et al.  MPI Applications on Grids: A Topology Aware Approach , 2009, Euro-Par.

[6]  Andreas Noack,et al.  Modularity clustering is force-directed layout , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[7]  Andrea Lancichinetti,et al.  Detecting the overlapping and hierarchical community structure in complex networks , 2008, 0802.1218.

[8]  Galen M. Shipman,et al.  MPI Support for Multi-core Architectures: Optimized Shared Memory Collectives , 2008, PVM/MPI.

[9]  Traian Muntean,et al.  Algorithms for Network Topology Discovery using End-to-End Measurements , 2008, 2008 International Symposium on Parallel and Distributed Computing.

[10]  Jean-Loup Guillaume,et al.  Fast unfolding of communities in large networks , 2008, 0803.0476.

[11]  Martin Rosvall,et al.  Maps of random walks on complex networks reveal community structure , 2007, Proceedings of the National Academy of Sciences.

[12]  Changjia Chen,et al.  A Rule-Based Traffic Exchange Matrix Estimation Algorithm for BitTorrent Tomography , 2006, 2006 8th international Conference on Signal Processing.

[13]  K. Doka,et al.  GridTorrent : Optimizing data transfers in the Grid with collaborative sharing , 2006 .

[14]  Robert Nowak,et al.  Network Tomography: Recent Developments , 2004 .

[15]  Mikel Izal,et al.  Dissecting BitTorrent: Five Months in a Torrent's Lifetime , 2004, PAM.

[16]  Robert D. Nowak,et al.  Multiple source, multiple destination network tomography , 2004, IEEE INFOCOM 2004.

[17]  M E J Newman,et al.  Finding and evaluating community structure in networks. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[18]  Emden R. Gansner,et al.  Graphviz and Dynagraph – Static and Dynamic Graph Drawing Tools , 2003 .

[19]  Ian T. Foster,et al.  MPICH-G2: A Grid-enabled implementation of the Message Passing Interface , 2002, J. Parallel Distributed Comput..

[20]  Martin Quinson,et al.  An Application-Level Network Mapper , 2003 .

[21]  B. Cohen,et al.  Incentives Build Robustness in Bit-Torrent , 2003 .

[22]  Helen J. Wang,et al.  Passive network tomography using Bayesian inference , 2002, IMW '02.

[23]  Pablo Rodriguez,et al.  Dynamic parallel access to replicated content in the internet , 2002, TNET.

[24]  Robert D. Nowak,et al.  Passive network tomography using EM algorithms , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[25]  Susumu Shibusawa,et al.  Scheduling algorithms for efficient gather operations in distributed heterogeneous systems , 2000, Proceedings 2000. International Workshop on Parallel Processing.

[26]  Henri E. Bal,et al.  Bandwidth-efficient collective communication for clustered wide area systems , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[27]  Donald F. Towsley,et al.  Multicast-based inference of network-internal characteristics: accuracy of packet loss estimation , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[28]  Michael M. Resch,et al.  Distributed Computing in a Heterogeneous Computing Environment , 1998, PVM/MPI.

[29]  Armin R. Mikler,et al.  NetPIPE: A Network Protocol Independent Performance Evaluator , 1996 .

[30]  Satoru Kawai,et al.  An Algorithm for Drawing General Undirected Graphs , 1989, Inf. Process. Lett..