Efficient and reliable network tomography in heterogeneous networks using BitTorrent broadcasts and clustering algorithms

In the area of network performance and discovery, network tomography focuses on reconstructing network properties using only end-to-end measurements at the application layer. One challenging problem in network tomography is reconstructing available bandwidth along all links during multiple source / multiple destination transmissions. The traditional measurement procedures used for bandwidth tomography are extremely time consuming. We propose a novel solution to this problem. Our method counts the fragments exchanged during a BitTorrent broadcast. While this measurement has a high level of randomness, it can be obtained very efficiently, and aggregated into a reliable metric. This data is then analyzed with state-of-the-art algorithms, which correctly reconstruct logical clusters of nodes interconnected by high bandwidth, as well as bottlenecks between these logical clusters. Our experiments demonstrate that the proposed two-phase approach efficiently solves the presented problem for a number of settings on a complex grid infrastructure.

[1]  Vijayan N. Nair,et al.  Statistical inverse problems in active network tomography , 2007, 0708.1079.

[2]  Donald F. Towsley,et al.  Multicast-based inference of network-internal loss characteristics , 1999, IEEE Trans. Inf. Theory.

[3]  Mikel Izal,et al.  Dissecting BitTorrent: Five Months in a Torrent's Lifetime , 2004, PAM.

[4]  Traian Muntean,et al.  Algorithms for Network Topology Discovery using End-to-End Measurements , 2008, 2008 International Symposium on Parallel and Distributed Computing.

[5]  Dimitrios Tsoumakos,et al.  Optimizing Data Management in Grid Environments , 2009, OTM Conferences.

[6]  Changjia Chen,et al.  A Rule-Based Traffic Exchange Matrix Estimation Algorithm for BitTorrent Tomography , 2006, 2006 8th international Conference on Signal Processing.

[7]  Andreas Noack,et al.  Modularity clustering is force-directed layout , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[8]  Helen J. Wang,et al.  Passive network tomography using Bayesian inference , 2002, IMW '02.

[9]  Pablo Rodriguez,et al.  Dynamic parallel access to replicated content in the internet , 2002, TNET.

[10]  Emden R. Gansner,et al.  Graphviz and Dynagraph - Static and Dynamic Graph Drawing Tools , 2004, Graph Drawing Software.

[11]  Jogesh K. Muppala,et al.  A Survey of BitTorrent Performance , 2010, IEEE Communications Surveys & Tutorials.

[12]  Armin R. Mikler,et al.  Net-PIPE: Network Protocol Independent Performance Evaluator , 1997 .

[13]  Thomas Hérault,et al.  MPI Applications on Grids: A Topology Aware Approach , 2009, Euro-Par.

[14]  Jack Dongarra,et al.  Recent Advances in Parallel Virtual Machine and Message Passing Interface, 15th European PVM/MPI Users' Group Meeting, Dublin, Ireland, September 7-10, 2008. Proceedings , 2008, PVM/MPI.

[15]  Martin Rosvall,et al.  Maps of random walks on complex networks reveal community structure , 2007, Proceedings of the National Academy of Sciences.

[16]  Donald F. Towsley,et al.  Multicast-based inference of network-internal characteristics: accuracy of packet loss estimation , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[17]  Sayantan Sur,et al.  Design and Evaluation of Network Topology-/Speed- Aware Broadcast Algorithms for InfiniBand Clusters , 2011, 2011 IEEE International Conference on Cluster Computing.

[18]  Satoru Kawai,et al.  An Algorithm for Drawing General Undirected Graphs , 1989, Inf. Process. Lett..

[19]  Songqing Chen,et al.  TopBT: A Topology-Aware and Infrastructure-Independent BitTorrent Client , 2010, 2010 Proceedings IEEE INFOCOM.

[20]  Jean-Loup Guillaume,et al.  Fast unfolding of communities in large networks , 2008, 0803.0476.

[21]  Robert D. Nowak,et al.  Passive network tomography using EM algorithms , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[22]  M E J Newman,et al.  Finding and evaluating community structure in networks. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[23]  Martin Quinson,et al.  An Application-Level Network Mapper , 2003 .

[24]  Robert Nowak,et al.  Network Tomography: Recent Developments , 2004 .

[25]  Henri Casanova,et al.  SimGrid: A Generic Framework for Large-Scale Distributed Experiments , 2008, Tenth International Conference on Computer Modeling and Simulation (uksim 2008).

[26]  Robert D. Nowak,et al.  Multiple source, multiple destination network tomography , 2004, IEEE INFOCOM 2004.

[27]  Henri E. Bal,et al.  Bandwidth-efficient collective communication for clustered wide area systems , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[28]  B. Cohen,et al.  Incentives Build Robustness in Bit-Torrent , 2003 .

[29]  Ian T. Foster,et al.  MPICH-G2: A Grid-enabled implementation of the Message Passing Interface , 2002, J. Parallel Distributed Comput..

[30]  Viktor K. Prasanna,et al.  Efficient collective communication in distributed heterogeneous systems , 2003, J. Parallel Distributed Comput..

[31]  Galen M. Shipman,et al.  MPI Support for Multi-core Architectures: Optimized Shared Memory Collectives , 2008, PVM/MPI.

[32]  Susumu Shibusawa,et al.  Scheduling algorithms for efficient gather operations in distributed heterogeneous systems , 2000, Proceedings 2000. International Workshop on Parallel Processing.

[33]  Andrea Lancichinetti,et al.  Detecting the overlapping and hierarchical community structure in complex networks , 2008, 0802.1218.

[34]  Benjamin H. Good,et al.  Performance of modularity maximization in practical contexts. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[35]  Michael M. Resch,et al.  Distributed Computing in a Heterogeneous Computing Environment , 1998, PVM/MPI.