Network traffic measurement for the next generation Internet

Measurement-based performance evaluation of network traffic is a fundamental prerequisite for the provisioning of managed and controlled services in short timescales, as well as for enabling the accountability of network resources. The steady introduction and deployment of the Internet Protocol Next Generation (IPNG-IPv6) promises a network address space that can accommodate any device capable of generating a digital heart-beat. Under such a ubiquitous communication environment, Internet traffic measurement becomes of particular importance, especially for the assured provisioning of differentiated levels of service quality to the different application flows. The non-identical response of flows to the different types of network-imposed performance degradation and the foreseeable expansion of networked devices raise the need for ubiquitous measurement mechanisms that can be equally applicable to different applications and transports. This thesis introduces a new measurement technique that exploits native features of IPv6 to become an integral part of the Internet's operation, and to provide intrinsic support for performance measurements at the universally-present network layer. IPv6 Extension Headers have been used to carry both the triggers that invoke the measurement activity and the instantaneous measurement indicators in-line with the payload data itself, providing a high level of confidence that the behaviour of the real user traffic flows is observed. The in-line measurements mechanism has been critically compared and contrasted to existing measurement techniques, and its design and a software-based prototype implementation have been documented. The developed system has been used to provisionally evaluate numerous performance properties of a diverse set of application flows, over different-capacity IPv6 experimental configurations. Through experimentation and theoretical argumentation, it has been shown that IPv6-based, in-line measurements can form the basis for accurate and low-overhead performance assessment of network traffic flows in short time-scales, by being dynamically deployed where and when required in a multi-service Internet environment.

[1]  Saverio Niccolini,et al.  Design and implementation of a one way delay passive measurement system , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[2]  Craig Partridge,et al.  A Proposed Flow Specification , 1992, RFC.

[3]  Dimitrios P. Pezaros,et al.  A native Measurement technique for IPv6-based Networks , 2004 .

[4]  Stephen E. Deering,et al.  Path MTU Discovery for IP version 6 , 1996, RFC.

[5]  Simon Leinen Evaluation of Candidate Protocols for IP Flow Information Export (IPFIX) , 2004, RFC.

[6]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[7]  David Hutchison,et al.  Quality of service assurance for the next generation Internet , 2001 .

[8]  David Hutchison,et al.  Service quality measurements for IPv6 inter-networks , 2004, Twelfth IEEE International Workshop on Quality of Service, 2004. IWQOS 2004..

[9]  David D. Clark,et al.  The design philosophy of the DARPA internet protocols , 1988, SIGCOMM '88.

[10]  Benjamin Teitelbaum,et al.  One-way Active Measurement Protocol (OWAMP) Requirements , 2004, RFC.

[11]  kc claffy,et al.  The nature of the beast: Recent traffic measurements from an Internet backbone , 1998 .

[12]  Geoff Huston Internet Performance Survival Guide , 2000 .

[13]  Donald F. Towsley,et al.  Estimation and removal of clock skew from network delay measurements , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[14]  Eric C. Rosen,et al.  Multiprotocol Label Switching Architecture , 2001, RFC.

[15]  Mats Björkman,et al.  A new end-to-end probing and analysis method for estimating bandwidth bottlenecks , 2000, Globecom '00 - IEEE. Global Telecommunications Conference. Conference Record (Cat. No.00CH37137).

[16]  Ori Pomerantz,et al.  The Linux Kernel Module Programming Guide , 2000 .

[17]  Nevil Brownlee,et al.  Traffic Flow Measurement: Experiences with NeTraMet , 1997, RFC.

[18]  kc claffy,et al.  Longitudinal study of Internet traffic in 1998-2003 , 2004 .

[19]  Al Morton,et al.  Standardized active measurements on a tier 1 IP backbone , 2003, IEEE Commun. Mag..

[20]  Keith McCloghrie,et al.  Protocol Operations for version 2 of the Simple Network Management Protocol (SNMPv2) , 1993, RFC.

[21]  Larry Peterson,et al.  Evaluating Network Processors in IP Forwarding , 2000 .

[22]  Konstantina Papagiannaki,et al.  Analysis of point-to-point packet delay in an operational network , 2007, Comput. Networks.

[23]  kc claffy,et al.  Application of sampling methodologies to network traffic characterization , 1993, SIGCOMM 1993.

[24]  Jean-Philipp Martin-Flatin Web-Based Management of IP Networks and Systems , 2002 .

[25]  Y. Vardi,et al.  Network Tomography: Estimating Source-Destination Traffic Intensities from Link Data , 1996 .

[26]  Vern Paxson,et al.  TCP Congestion Control , 1999, RFC.

[27]  Matthew J. Luckie,et al.  Towards improving packet probing techniques , 2001, IMW '01.

[28]  David L. Black,et al.  An Architecture for Differentiated Service , 1998 .

[29]  Sally Floyd,et al.  TCP Selective Acknowledgement Options , 1996 .

[30]  V. Jacobson,et al.  Congestion avoidance and control , 1988, CCRV.

[31]  Anja Feldmann,et al.  NetScope: traffic engineering for IP networks , 2000, IEEE Netw..

[32]  David Hutchison,et al.  In-line service measurements: an IPv6-based framework for traffic evaluation and network operations , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[33]  Anja Feldmann,et al.  IP network configuration for intradomain traffic engineering , 2001, IEEE Netw..

[34]  George Varghese,et al.  Building a better NetFlow , 2004, SIGCOMM 2004.

[35]  Peter B. Danzig,et al.  Characteristics of wide-area TCP/IP conversations , 1991, SIGCOMM 1991.

[36]  Ian Graham,et al.  Design principles for accurate passive measurement , 2000 .

[37]  D. O. Awduche,et al.  MPLS and traffic engineering in IP networks , 1999, IEEE Commun. Mag..

[38]  kc claffy,et al.  The architecture of CoralReef: an Internet traffic monitoring software suite , 2001 .

[39]  Dimitrios P. Pezaros In-line Service Measurements: Instrumenting the IPv6 Internet , 2003 .

[40]  Zheng Wang,et al.  An Architecture for Differentiated Services , 1998, RFC.

[41]  Vern Paxson,et al.  An architecture for large-scale Internet measurement , 1998, IEEE Commun. Mag..

[42]  Bert Wijnen,et al.  An Architecture for Describing Simple Network Management Protocol (SNMP) Management Frameworks , 2002, RFC.

[43]  Hans-Werner Braun,et al.  The NLANR network analysis infrastructure , 2000, IEEE Commun. Mag..

[44]  Geoff Huston,et al.  Quality of Service on the Internet: Fact, Fiction, or Compromise? , 2003 .

[45]  Dimitrios P. Pezaros Measurement, Monitoring and Control of Internet Traffic for Improved QoS , 2002 .

[46]  Sally Floyd,et al.  The NewReno Modification to TCP's Fast Recovery Algorithm , 2004, RFC.

[47]  Anja Feldmann,et al.  Using the AT&T Labs PacketScope for Internet Measurement, Design, and Performance Analysis , 1997 .

[48]  M. Alves New measurements with the ripe ncc test traffic measurements setup , 2002 .

[49]  Sebastian Abeck,et al.  Integrated Management of Networked Systems: Concepts, Architectures and their Operational Application , 1999 .

[50]  Walter Willinger,et al.  On the self-similar nature of Ethernet traffic , 1993, SIGCOMM '93.

[51]  Bryan Henderson Linux Loadable Kernel Module HOWTO , 2006 .

[52]  Stephen T. Kent,et al.  IP Authentication Header , 1995, RFC.

[53]  Randall J. Atkinson,et al.  IP Encapsulating Security Payload (ESP) , 1995, RFC.

[54]  Donald F. Towsley,et al.  Modeling TCP throughput: a simple model and its empirical validation , 1998, SIGCOMM '98.

[55]  Sandrine Vaton,et al.  Network tomography: an iterative Bayesian analysis , 2003 .

[56]  Sally Floyd,et al.  Promoting the use of end-to-end congestion control in the Internet , 1999, TNET.

[57]  Chase Cotton,et al.  Packet-level traffic measurements from the Sprint IP backbone , 2003, IEEE Netw..

[58]  Matthew Mathis,et al.  The macroscopic behavior of the TCP congestion avoidance algorithm , 1997, CCRV.

[59]  kc claffy,et al.  Bandwidth estimation: metrics, measurement techniques, and tools , 2003, IEEE Netw..

[60]  Brian E. Carpenter,et al.  IPv6 Flow Label Specification , 2004, RFC.

[61]  Van Jacobson,et al.  An Expedited Forwarding PHB , 1999, RFC.

[62]  Paul Barford,et al.  Modeling Congestion in Backbone Routers , 2002 .

[63]  William Stallings,et al.  SNMP, SNMPv2, SNMPv3, and RMON 1 and 2 , 1999 .

[64]  F.J. Garcia,et al.  Inline measurements: a native measurement technique for IPv6 networks , 2004, 2004 International Networking and Communication Conference.

[65]  Mats Björkman,et al.  Regression-Based Available Bandwidth Measurements , 2002 .

[66]  Manish Jain,et al.  End-to-end available bandwidth: measurement methodology, dynamics, and relation with TCP throughput , 2002, SIGCOMM 2002.

[67]  Brian Tierney,et al.  System capability effects on algorithms for network bandwidth measurement , 2003, IMC '03.

[68]  Matt Thomas,et al.  Advanced Sockets API for IPv6 , 1998, RFC.

[69]  Matthew J. Zekauskas,et al.  A One-way Delay Metric for IPPM , 1999, RFC.

[70]  Sidnie M. Feit SNMP: A Guide to Network Management , 1993 .

[71]  Nevil Brownlee SRL: A Language for Describing Traffic Flows and Specifying Actions for Flow Groups , 1999, RFC.

[72]  Anil Rijsinghani,et al.  Computation of the Internet Checksum via Incremental Update , 1994, RFC.

[73]  Luca Deri,et al.  Effective traffic measurement using ntop , 2000 .

[74]  Cristian Estan,et al.  New directions in traffic measurement and accounting , 2001, IMW '01.

[75]  Vern Paxson,et al.  On calibrating measurements of packet transit times , 1998, SIGMETRICS '98/PERFORMANCE '98.

[76]  Paul Barford,et al.  Measuring Web performance in the wide area , 1999, PERV.

[77]  Lillian N. Cassel,et al.  Management of sampled real-time network measurements , 1989, [1989] Proceedings. 14th Conference on Local Computer Networks.

[78]  Anja Feldmann,et al.  On the impact of variability on the buffer dynamics in IP networks , 1999 .

[79]  Philip F. Chimento,et al.  IP Packet Delay Variation Metric for IP Performance Metrics (IPPM) , 2002, RFC.

[80]  Scott O. Bradner,et al.  The Recommendation for the IP Next Generation Protocol , 1995, RFC.

[81]  Ulf Nordqvist Protocol Processing in Network Terminals , 2004 .

[82]  William Stallings SNMP, SNMPv2, and RMON: Practical Network Management , 1996 .

[83]  Allen B. Downey Using pathchar to estimate Internet link characteristics , 1999, SIGCOMM '99.

[84]  Olivier Bonaventure,et al.  Using Redistribution Communities for Interdomain Traffic Engineering , 2002, QofIS.

[85]  Mark Allman,et al.  A Framework for Defining Empirical Bulk Transfer Capacity Metrics , 2001, RFC.

[86]  George Varghese,et al.  The measurement manifesto , 2004, Comput. Commun. Rev..

[87]  Vern Paxson,et al.  IPPM Metrics for Measuring Connectivity , 1999, RFC.

[88]  Benjamin Teitelbaum,et al.  A One-way Active Measurement Protocol (OWAMP) , 2006, RFC.

[89]  David Hutchison,et al.  Dynamic Link Measurements Using Active Components , 2004, IWAN.

[90]  Roch Guérin,et al.  Achieving near-optimal traffic engineering solutions for current OSPF/IS-IS networks , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[91]  Joe Finney,et al.  Component-based active network architecture , 2001, Proceedings. Sixth IEEE Symposium on Computers and Communications.

[92]  Darryl Veitch,et al.  PC based precision timing without GPS , 2002, SIGMETRICS '02.

[93]  David D. Clark,et al.  A knowledge plane for the internet , 2003, SIGCOMM '03.

[94]  Matthew J. Zekauskas,et al.  A Round-trip Delay Metric for IPPM , 1999, RFC.

[95]  George C. Polyzos,et al.  Traffic characteristics of the T1 NSFNET backbone , 1993, IEEE INFOCOM '93 The Conference on Computer Communications, Proceedings.

[96]  Sally Floyd,et al.  Connections with multiple congested gateways in packet-switched networks part 1: one-way traffic , 1991, CCRV.

[97]  한선영 [서평]「Understanding Networked Multimedia applications and technology」 , 1996 .

[98]  Nevil Brownlee,et al.  RTFM: New Attributes for Traffic Flow Measurement , 1999, RFC.

[99]  Jeffrey D. Case,et al.  Simple Network Management Protocol (SNMP) , 1989, RFC.

[100]  Matthew J. Zekauskas,et al.  A One-way Packet Loss Metric for IPPM , 1999, RFC.

[101]  Douglas Comer,et al.  Internetworking with TCP/IP , 1988 .

[102]  Steven Waldbusser Remote Network Monitoring Management Information Base , 1991, RFC.

[103]  Lillian N. Cassel,et al.  Local Area Network Broadcast Measurement: Traffic Characterization , 1987, IEEE Computer Society International Conference.

[104]  Parameswaran Ramanathan,et al.  Packet-dispersion techniques and a capacity-estimation methodology , 2004, IEEE/ACM Transactions on Networking.

[105]  Henning Schulzrinne,et al.  RTP: A Transport Protocol for Real-Time Applications , 1996, RFC.

[106]  Vern Paxson,et al.  Towards a Framework for Defining Internet Performance Metrics , 2006 .

[107]  Anja Feldmann,et al.  Deriving traffic demands for operational IP networks: methodology and experience , 2001, TNET.

[108]  Rajeev Koodli,et al.  One-way Loss Pattern Sample Metrics , 2002, RFC.

[109]  Craig Partridge,et al.  Technical Criteria for Choosing IP The Next Generation (IPng) , 1994, RFC.

[110]  Murad S. Taqqu,et al.  On the Self-Similar Nature of Ethernet Traffic , 1993, SIGCOMM.

[111]  Vern Paxson,et al.  Framework for IP Performance Metrics , 1998, RFC.

[112]  Murray Pearson,et al.  High precision traffic measurement , 2002 .

[113]  Tobias Oetiker,et al.  MRTG: The Multi Router Traffic Grapher , 1998, LISA.

[114]  Scott Shenker,et al.  Integrated Services in the Internet Architecture : an Overview Status of this Memo , 1994 .

[115]  Klara Nahrstedt,et al.  Multimedia: Computing, Communications and Applications , 1994 .

[116]  Robert T. Braden,et al.  Requirements for Internet Hosts - Communication Layers , 1989, RFC.

[117]  Ramón Cáceres Measurements of Wide Area Internet TraffiC , 1989 .

[118]  Robert E. Gilligan,et al.  Basic Socket Interface Extensions for IPv6 , 1997, RFC.

[119]  Stephen Donnelly,et al.  Passive Calibration of an Active Measurement System , 2001 .

[120]  Douglas E. Comer,et al.  Internetworking with TCP/IP - Principles, Protocols, and Architectures, Fourth Edition , 1988 .

[121]  Christophe Diot,et al.  Traffic matrix estimation: existing techniques and new directions , 2002, SIGCOMM 2002.

[122]  Marshall T. Rose,et al.  Management Information Base for network management of TCP/IP-based internets , 1990, RFC.

[123]  Evan McGinnis,et al.  Understanding SNMP MIBs , 1996 .

[124]  Vern Paxson,et al.  Measurements and analysis of end-to-end Internet dynamics , 1997 .

[125]  Ian Wakeman,et al.  Traffic analysis of some uk-us academic network data , 1991 .

[126]  Constantinos Dovrolis,et al.  The effect of layer-2 store-and-forward devices on per-hop capacity estimation , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[127]  Konstantina Papagiannaki,et al.  Analysis of point-to-point packet delay in an operational network , 2004, IEEE INFOCOM 2004.

[128]  Johan A. Pouwelse,et al.  The Bittorrent P2P File-Sharing System: Measurements and Analysis , 2005, IPTPS.

[129]  Al Morton,et al.  Network performance measurement with periodic streams , 2002, RFC.

[130]  Fabrice Guillemin,et al.  Analysis of ADSL traffic on an IP backbone link , 2003, GLOBECOM '03. IEEE Global Telecommunications Conference (IEEE Cat. No.03CH37489).

[131]  Robert Beverly RTG: A Scalable SNMP Statistics Architecture for Service Providers , 2002, LISA.

[132]  R. Les Cottrell,et al.  PingER History and Methodology , 2003 .

[133]  Stephen E. Deering,et al.  Internet Protocol, Version 6 (IPv6) Specification , 1995, RFC.

[134]  W. Richard Stevens,et al.  TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms , 1997, RFC.

[135]  Keith McCloghrie,et al.  Structure and identification of management information for TCP/IP-based internets , 1988, RFC.

[136]  Nevil Brownlee Using NeTraMet for production traffic measurement , 2001, 2001 IEEE/IFIP International Symposium on Integrated Network Management Proceedings. Integrated Network Management VII. Integrated Management Strategies for the New Millennium (Cat. No.01EX470).

[137]  Nick Duffield,et al.  Sampling for Passive Internet Measurement: A Review , 2004 .

[138]  Manish Jain,et al.  End-to-end available bandwidth: measurement methodology, dynamics, and relation with TCP throughput , 2003, IEEE/ACM Trans. Netw..

[139]  Philippe Owezarski,et al.  Design and Deployment of a Passive Monitoring Infrastructure , 2001, IWDC.

[140]  George C. Polyzos,et al.  Measurement Considerations for Assessing Unidirectional Latencies � , 1993 .

[141]  Anja Feldmann,et al.  Measurement and analysis of IP network usage and behavior , 2000, IEEE Commun. Mag..

[142]  David L. Mills,et al.  Internet time synchronization: the network time protocol , 1991, IEEE Trans. Commun..

[143]  Scott O. Bradner,et al.  IP: Next Generation (IPng) White Paper Solicitation , 1993, RFC.

[144]  Raj Jain,et al.  Packet Trains-Measurements and a New Model for Computer Network Traffic , 1986, IEEE J. Sel. Areas Commun..

[145]  W. Richard Stevens,et al.  UNIX network programming, 2nd Edition , 1998 .

[146]  W. Matthews,et al.  Internet end-to-end performance monitoring for the High Energy Nuclear and Particle Physics community , 2000 .

[147]  Gary Scott Malkin,et al.  Traceroute Using an IP Option , 1993, RFC.

[148]  Jürgen Quittek,et al.  Requirements for IP Flow Information Export (IPFIX) , 2004, RFC.

[149]  Aiko Pras,et al.  Internet management: status and challenges , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[150]  Steven McCanne,et al.  The BSD Packet Filter: A New Architecture for User-level Packet Capture , 1993, USENIX Winter.

[151]  Kimberly C. Claffy,et al.  OC3MON: Flexible, Affordable, High Performance Staistics Collection , 1996, LISA.

[152]  Bing Yu,et al.  Time-Varying Network Tomography: Router Link Data , 2000 .

[153]  Les Cottrell,et al.  The PingER project: active Internet performance monitoring for the HENP community , 2000, IEEE Commun. Mag..

[154]  Nevil Brownlee,et al.  Traffic Flow Measurement: Architecture , 1999, RFC.

[155]  Kevin Jeffay,et al.  Application-level measurements of performance on the vBNS , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[156]  Jennifer Rexford,et al.  Passive Traac Measurement for Ip Operations , 2002 .

[157]  Hans-Werner Braun,et al.  Internet Flow Characterization: Adaptive Timeout Strategy and Statistical Modeling , 2001 .

[158]  Nevil Brownlee,et al.  Traffic Flow Measurement: Meter MIB , 1997, RFC.

[159]  kc claffy,et al.  Understanding Internet traffic streams: dragonflies and tortoises , 2002, IEEE Commun. Mag..

[160]  Sunil Kalidindi,et al.  Surveyor: An Infrastructure For Internet Performance Measurements , 1999 .

[161]  Steven Waldbusser Remote Network Monitoring Management Information Base Version 2 using SMIv2 , 1997, RFC.