BLT: Bi-Layer Tracing of HTTP and TCP/IP

Abstract We describe BLT, a tool for extracting full HTTP level as well as TCP level traces via packet monitoring. This paper presents the software architecture that allows us to collect traces continuously, online, and at any point in the network. The software has been used to extract extensive traces within AT&T WorldNet since spring 1997 as well as at AT&T Labs-Research. BLT offers a much richer alternative to Web proxy logs, client logs, and Web server logs due to the accuracy of the timestamps, the level of details available by considering several protocol layers (TCP/IP and HTTP events), and its non intrusive way of gathering data. Traces gathered using BLT have provided the foundation of several Web performance studies.

[1]  Balachander Krishnamurthy,et al.  Study of Piggyback Cache Validation for Proxy Caches in the World Wide Web , 1997, USENIX Symposium on Internet Technologies and Systems.

[2]  Farnam Jahanian,et al.  An extensible probe architecture for network protocol performance measurement , 1998, SIGCOMM '98.

[3]  Willy Zwaenepoel,et al.  Efficient Support for P-HTTP in Cluster-Based Web Servers , 1999, USENIX Annual Technical Conference, General Track.

[4]  George C. Polyzos,et al.  A Parameterizable Methodology for Internet Traffic Flow Profiling , 1995, IEEE J. Sel. Areas Commun..

[5]  Paul Barford,et al.  Changes in Web Client Access Patterns , 1998, The Web Conference.

[6]  Darrell D. E. Long,et al.  Exploring the Bounds of Web Latency Reduction from Caching and Prefetching , 1997, USENIX Symposium on Internet Technologies and Systems.

[7]  Balachander Krishnamurthy,et al.  PRO-COW: Protocol Compliance on the Web , 1999 .

[8]  Rick Wilder,et al.  Wide-Area Internet Traffic Patterns and Characteristics ( Extended Version ) , 1997 .

[9]  Deborah Estrin,et al.  An Empirical Workload Model for Driving Wide-Area TCP/IP Network Simulations , 2001 .

[10]  Azer Bestavros,et al.  Self-similarity in World Wide Web traffic: evidence and possible causes , 1996, SIGMETRICS '96.

[11]  Anja Feldmann,et al.  Rate of Change and other Metrics: a Live Study of the World Wide Web , 1997, USENIX Symposium on Internet Technologies and Systems.

[12]  Mark Crovella,et al.  Self-Similarity in World Wide Web Traffic: Evidence and Causes , 1996, SIGMETRICS.

[13]  Srinivasan Seshan,et al.  TCP behavior of a busy Internet server: analysis and improvements , 1997, Proceedings. IEEE INFOCOM '98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies. Gateway to the 21st Century (Cat. No.98.

[14]  Anja Feldmann,et al.  Web proxy caching: the devil is in the details , 1998, PERV.

[15]  Sally Floyd,et al.  Wide area traffic: the failure of Poisson modeling , 1995, TNET.

[16]  Edith Cohen,et al.  Improving end-to-end performance of the Web using server volumes and proxy filters , 1998, SIGCOMM '98.

[17]  Margo I. Seltzer,et al.  Web Facts and Fantasy , 1997, USENIX Symposium on Internet Technologies and Systems.

[18]  K. K. Ramakrishnan,et al.  Eliminating receive livelock in an interrupt-driven kernel , 1996, TOCS.

[19]  Jeffrey C. Mogul,et al.  Improving HTTP Latency , 1995, Comput. Networks ISDN Syst..

[20]  Balachander Krishnamurthy,et al.  Piggyback Server Invalidation for Proxy Cache Coherency , 1998, Comput. Networks.

[21]  Anja Feldmann,et al.  Performance of Web proxy caching in heterogeneous bandwidth environments , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[22]  Paul Barford,et al.  Generating representative Web workloads for network and server performance evaluation , 1998, SIGMETRICS '98/PERFORMANCE '98.

[23]  Anja Feldmann,et al.  Using the AT&T Labs PacketScope for Internet Measurement, Design, and Performance Analysis , 1997 .

[24]  Carey L. Williamson,et al.  Internet Web servers: workload characterization and performance implications , 1997, TNET.

[25]  Anja Feldmann,et al.  The changing nature of network traffic: scaling phenomena , 1998, CCRV.

[26]  Anja Feldmann,et al.  Potential benefits of delta encoding and data compression for HTTP , 1997, SIGCOMM '97.

[27]  Debanjan Saha,et al.  L5: A Self Learning Layer-5 Switch , 1999 .

[28]  Larry Wall,et al.  Programming Perl , 1991 .

[29]  Eric A. Brewer,et al.  System Design Issues for Internet Middleware Services: Deductions from a Large Client Trace , 1997, USENIX Symposium on Internet Technologies and Systems.

[30]  Anja Feldmann,et al.  Reducing overhead in flow-switched networks: an empirical study of Web traffic , 1998, Proceedings. IEEE INFOCOM '98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies. Gateway to the 21st Century (Cat. No.98.