Performance analysis and comparison of interrupt-handling schemes in gigabit networks

Interrupt processing can be a major bottleneck in the end-to-end performance of Gigabit networks. The performance of Gigabit network end hosts or servers can be severely degraded due to interrupt overhead caused by heavy incoming traffic. In particular, excessive latency and significant degradation in system throughput can be encountered. Also, user applications may livelock as the CPU power gets mostly consumed by interrupt handling and protocol processing. A number of interrupt-handling schemes has been proposed and employed to mitigate the interrupt overhead and improve OS performance. Among the most popular interrupt-handling schemes are normal interruption, polling, interrupt coalescing, and disabling and enabling of interrupts. In previous work, we presented a preliminary analytical study and models of normal interruption and interrupt coalescing. In this article, we extend our analysis and modeling to include polling and the scheme of interrupt disabling and enabling. For polling, we study both pure (or FreeBSD-style) polling and Linux NAPI polling. The performances for all these schemes are compared using both mathematical analysis and discrete-event simulation. The performance is studied in terms of three key performance indicators: throughput, system latency, and the residual CPU bandwidth available for user applications. As opposed to our previous work, we consider not only Poisson traffic, but also bursty traffic with empirical packet size distribution. Our analysis and simulation work gives insight into predicting the system performance and behavior when employing a certain interrupt-handling scheme. It is concluded that no single interrupt-handling scheme outperforms all other schemes under all traffic conditions. Based on obtained results, we propose and discuss a novel hybrid scheme of interrupt disabling-enabling and pure polling in order to attain peak performance under low and heavy traffic loads.

[1]  K. K. Ramakrishnan,et al.  Performance Considerations in Designing Network Interfaces , 1993, IEEE J. Sel. Areas Commun..

[2]  K. K. Ramakrishnan,et al.  Eliminating receive livelock in an interrupt-driven kernel , 1996, TOCS.

[3]  Eddie Kohler,et al.  The Click modular router , 1999, SOSP.

[4]  Parameswaran Ramanathan,et al.  HIP: hybrid interrupt-polling for the network interface , 2001, OPSR.

[5]  Carl Staelin,et al.  lmbench: Portable Tools for Performance Analysis , 1996, USENIX Annual Technical Conference.

[6]  Khaled Salah,et al.  Throughput and delay analysis of interrupt-driven kernels under Poisson and bursty traffic , 2007, Comput. Syst. Sci. Eng..

[7]  Walter Willinger,et al.  Self-Similar Network Traffic and Performance Evaluation , 2000 .

[8]  Jamal Hadi Salim,et al.  Beyond Softnet , 2001, Annual Linux Showcase & Conference.

[9]  Donald F. Towsley,et al.  Inferring network characteristics via moment-based estimators , 2001, Proceedings IEEE INFOCOM 2001. Conference on Computer Communications. Twentieth Annual Joint Conference of the IEEE Computer and Communications Society (Cat. No.01CH37213).

[10]  Ronald W. Wolff,et al.  Efficient simulation of queues in heavy traffic , 2003, TOMC.

[11]  Peter Druschel,et al.  Soft timers: efficient microsecond software timer support for network processing , 1999, SOSP.

[12]  Henri E. Bal,et al.  User-Level Network Interface Protocols , 1998, Computer.

[13]  O. Brun,et al.  Analytical solution of finite capacity M/D/1 queues , 2000, Journal of Applied Probability.

[14]  Steven G. Johnson,et al.  The Design and Implementation of FFTW3 , 2005, Proceedings of the IEEE.

[15]  Miljenko Mikuc,et al.  Estimating the Impact of Interrupt Coalescing Delays on Steady State TCP Throughput , 2002 .

[16]  Hsiao-Keng Jerry Chu,et al.  Zero-Copy TCP in Solaris , 1996, USENIX Annual Technical Conference.

[17]  Khaled Salah.  To coalesce or not to coalesce , 2007 .

[18]  Fouad A. Tobagi,et al.  Analysis of delay and delay jitter of voice traffic in the Internet , 2002, Comput. Networks.

[19]  Jonathan S. Shapiro,et al.  USENIX Association Proceedings of the General Track : 2004 USENIX Annual Technical Conference , 2004 .

[20]  Manish Jain,et al.  Effects of Interrupt Coalescence on Network Measurements , 2004, PAM.

[21]  Lester Lipsky,et al.  Long-lasting transient conditions in simulations with heavy-tailed workloads , 1997, WSC '97.

[22]  P. Wyckoff,et al.  EMP: Zero-Copy OS-Bypass NIC-Driven Gigabit Ethernet Message Passing , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[23]  K. Preston White,et al.  An Effective Truncation Heuristic for Bias Reduction in Simulation Output , 1997, Simul..

[24]  Peter Druschel,et al.  Operating system support for high-speed communication , 1996, CACM.

[25]  Andrea Bianco,et al.  Open-Source PC-Based Software Routers: A Viable Approach to High-Performance Packet Switching , 2005, QoS-IP.

[26]  Sally Floyd,et al.  Wide-area traffic: the failure of Poisson modeling , 1994 .

[27]  Khaled Salah,et al.  Evaluating system performance in Gigabit networks , 2003, 28th Annual IEEE International Conference on Local Computer Networks, 2003. LCN '03. Proceedings..

[28]  Jonathan M. Smith,et al.  Hardware/Software Organization of a High-Performance ATM Host Interface , 1993, IEEE J. Sel. Areas Commun..

[29]  Guru M. Parulkar,et al.  The APIC approach to high performance network interface design: protected DMA and other techniques , 1997, Proceedings of INFOCOM '97.

[30]  J.M. Smith,et al.  Giving applications access to Gb/s networking , 1993, IEEE Network.

[31]  Khaled Salah,et al.  Modeling and Analysis of Interrupt Disable-Enable Scheme , 2007, 21st International Conference on Advanced Information Networking and Applications (AINA '07).

[32]  L. Deri.  Improving Passive Packet Capture : Beyond Device Polling , 2003 .

[33]  Jeffrey S. Chase,et al.  End system optimizations for high-speed TCP , 2001, IEEE Commun. Mag..

[34]  H.H.J. Hum,et al.  Polling Watchdog: Combining Polling and Interrupts for Efficient Message Handling , 1996, 23rd Annual International Symposium on Computer Architecture (ISCA'96).

[35]  David D. Clark,et al.  An analysis of TCP processing overhead , 1988, IEEE Communications Magazine.

[36]  EDDIE KOHLER,et al.  The click modular router , 2000, TOCS.

[37]  Walter Willinger,et al.  On the Self-Similar Nature of Ethernet Traffic ( extended version ) , 1995 .

[38]  Rob Williams,et al.  Linux device drivers , 2006 .

[39]  Jeffrey S. Chase,et al.  Trapeze / IP : TCP / IP at Near-Gigabit Speeds , 1999 .

[40]  Khaled Salah,et al.  Performance evaluation of interrupt-driven kernels in gigabit networks , 2003, GLOBECOM '03. IEEE Global Telecommunications Conference (IEEE Cat. No.03CH37489).

[41]  Samuel J. Leffler,et al.  The design and implementation of the 4.3 BSD Unix operating system , 1991, Addison-Wesley series in computer science.

[42]  Walter Willinger,et al.  Proof of a fundamental result in self-similar traffic modeling , 1997, CCRV.

[43]  Peter Druschel,et al.  Lazy receiver processing (LRP): a network subsystem architecture for server systems , 1996, OSDI '96.

[44]  Walter Willinger,et al.  Self-similarity through high-variability: statistical analysis of Ethernet LAN traffic at the source level , 1997, TNET.

[45]  José Carlos Brustoloni,et al.  Effects of buffering semantics on I/O performance , 1996, OSDI '96.

[46]  Greg J. Regnier,et al.  TCP performance re-visited , 2003, 2003 IEEE International Symposium on Performance Analysis of Systems and Software. ISPASS 2003..

[47]  kc claffy,et al.  The nature of the beast: Recent traffic measurements from an Internet backbone , 1998 .

[48]  Walter Willinger,et al.  On the self-similar nature of Ethernet traffic , 1993, SIGCOMM '93.

[49]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[50]  K. El-Badawi,et al.  Analysis and simulation of interrupt overhead impact on OS throughput in high‐speed networks , 2005, Int. J. Commun. Syst..

[51]  Margo I. Seltzer,et al.  Operating system benchmarking in the wake of lmbench: a case study of the performance of NetBSD on the Intel x86 architecture , 1997, SIGMETRICS '97.

[52]  Ulrich Killat,et al.  How many traffic sources are enough? , 2004 .