Computer Sciences Department Understanding and Exploiting Network Traffic Redundancy

Abstract—The Internet carries a vast amount and a wide rangeof content. Some of this content is more popular, and accessedmore frequently, than others. The popularity of content couldbe quite ephemeral - e.g., a Web flash crowd - or much morepermanent - e.g., google.com’s banner. A direct consequence ofthe skew in popularity is that, at any time, a fraction of theinformation carried over the Internet is redundant.We make two contributions in this paper. First, we study thefundamental properties of the redundancy in the informationcarried over the Internet, with a focus on network edges. Wecollect traffic traces at two network edge locations – a largeuniversity’s access link serving roughly 50,000 users, and a tier-1ISP network link connected to a large data center. We conductseveral analyses over this data: What fraction of bytes areredundant? What is the frequency at which strings of bytes re peatacross different packets? What is the overlap in the informa tionaccessed by distinct groups of end-users?Second, we leverage our measurement observations in thedesign of a family mechanisms for eliminating redundancy innetwork traffic and improving the overall network performan ce.The mechanisms we proposed can improve the available capacityof single network links as well as balance load across multiplenetwork links.