ELI: bare-metal performance for I/O virtualization

Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because the host intercepts all interrupts, including those interrupts generated by assigned devices to signal to guests the completion of their I/O requests. The host involvement induces multiple unwarranted guest/host context switches, which significantly hamper the performance of I/O intensive workloads. To solve this problem, we present ELI (ExitLess Interrupts), a software-only approach for handling interrupts within guest virtual machines directly and securely. By removing the host from the interrupt handling path, ELI manages to improve the throughput and latency of unmodified, untrusted guests by 1.3x-1.6x, allowing them to reach 97%-100% of bare-metal performance even for the most demanding I/O-intensive workloads.

[1]  Willy Zwaenepoel,et al.  Diagnosing performance overheads in the xen virtual machine environment , 2005, VEE '05.

[2]  K. K. Ramakrishnan,et al.  Eliminating receive livelock in an interrupt-driven kernel , 1996, TOCS.

[3]  Radu Rugina,et al.  Software Techniques for Avoiding Hardware Virtualization Exits , 2012, USENIX Annual Technical Conference.

[4]  Assaf Schuster,et al.  MultiView and Millipage — fine-grain sharing in page-based DSMs , 1999, OSDI '99.

[5]  Daniel Pierre Bovet,et al.  Understanding the Linux Kernel , 2000 .

[6]  G. Cox,et al.  ~ " " " ' l I ~ " " -" . : -· " J , 2006 .

[7]  Miljenko Mikuc,et al.  Estimating the Impact of Interrupt Coalescing Delays on Steady State TCP Throughput , 2002 .

[8]  Muli Ben-Yehuda,et al.  SplitX: Split Guest/Hypervisor Execution on Multi-Core , 2011, WIOV.

[9]  Muli Ben-Yehuda,et al.  Adding advanced storage controller functionality via low-overhead virtualization , 2012, FAST.

[10]  Xiaowei Yang,et al.  High performance network virtualization with SR-IOV , 2010, HPCA - 16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture.

[11]  Stefan Götz,et al.  Unmodified Device Driver Reuse and Improved System Dependability via Virtual Machines , 2004, OSDI.

[12]  Rusty Russell,et al.  virtio: towards a de-facto standard for virtual I/O devices , 2008, OPSR.

[13]  Dhabaleswar K. Panda,et al.  High Performance VMM-Bypass I/O in Virtual Machines , 2006, USENIX Annual Technical Conference, General Track.

[14]  Gerald J. Popek,et al.  Formal requirements for virtualizable third generation architectures , 1974, SOSP '73.

[15]  Ram Huggahalli,et al.  Architectural Breakdown of End-to-End Latency in a TCP/IP Network , 2007, 19th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD'07).

[16]  Jennifer Rexford,et al.  Eliminating the hypervisor attack surface for a more secure cloud , 2011, CCS '11.

[17]  Peter Druschel,et al.  Transparent operating system support for superpages , 2004 .

[18]  Zhao Yu,et al.  SR-IOV Networking in Xen: Architecture, Design and Implementation , 2008, Workshop on I/O Virtualization.

[19]  Gil Neiger,et al.  Intel ® Virtualization Technology for Directed I/O , 2006 .

[20]  Brad Fitzpatrick,et al.  Distributed caching with memcached , 2004 .

[21]  Jose Renato Santos,et al.  Bridging the Gap between Software and Hardware Techniques for I/O Virtualization , 2008, USENIX Annual Technical Conference.

[22]  No License,et al.  Intel ® 64 and IA-32 Architectures Software Developer ’ s Manual Volume 3 A : System Programming Guide , Part 1 , 2006 .

[23]  Ole Agesen,et al.  A comparison of software and hardware techniques for x86 virtualization , 2006, ASPLOS XII.

[24]  Vishakha Gupta,et al.  High-Performance Hypervisor Architectures: Virtualization in HPC Systems , 2007 .

[25]  Rafal Wojtczuk,et al.  Following the White Rabbit : Software attacks against Intel ( R ) VT-d technology , 2011 .

[26]  Jennifer Rexford,et al.  NoHype: virtualized cloud infrastructure without the virtualization , 2010, ISCA.

[27]  Peter A. Dinda,et al.  Minimal-overhead virtualization of a large scale supercomputer , 2011, VEE '11.

[28]  Yang Zhang,et al.  Optimizing Network I/O Virtualization with Efficient Interrupt Coalescing and Virtual Receive Side Scaling , 2011, 2011 IEEE International Conference on Cluster Computing.

[29]  Alan L. Cox,et al.  Practical, transparent operating system support for superpages , 2002, OPSR.

[30]  A. Kivity,et al.  kvm : the Linux Virtual Machine Monitor , 2007 .

[31]  Khaled Salah.  To coalesce or not to coalesce , 2007 .

[32]  Dan Tsafrir,et al.  System noise, OS clock ticks, and fine-grained parallel applications , 2005, ICS '05.

[33]  Alan L. Cox,et al.  Concurrent Direct Network Access for Virtual Machine Monitors , 2007, 2007 IEEE 13th International Symposium on High Performance Computer Architecture.

[34]  Beng-Hong Lim,et al.  Virtualizing I/O Devices on VMware Workstation's Hosted Virtual Machine Monitor , 2001, USENIX Annual Technical Conference, General Track.

[35]  Irfan Ahmad,et al.  vIC: Interrupt Coalescing for Virtual Machine Storage Device IO , 2011, USENIX Annual Technical Conference.

[36]  K. Salah,et al.  Boosting throughput of Snort NIDS under Linux , 2008, 2008 International Conference on Innovations in Information Technology.

[37]  Karsten Schwan,et al.  High performance and scalable I/O virtualization via self-virtualized devices , 2007, HPDC '07.

[38]  Gil Neiger,et al.  Intel virtualization technology , 2005, Computer.

[39]  Jiuxing Liu.  Evaluating standard-based self-virtualizing devices: A performance study on 10 GbE NICs with SR-IOV support , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[40]  Laxmi N. Bhuyan,et al.  Software techniques to improve virtualized I/O performance on multi-core systems , 2008, ANCS '08.

[41]  Hari Angepat,et al.  Reflective Virtualization Improving the Performance of Fully-Virtualized x 86 Operating Systems , 2007 .

[42]  Alan L. Cox,et al.  Achieving 10 Gb/s using safe and transparent network interface virtualization , 2009, VEE '09.

[43]  Muli Ben-Yehuda,et al.  vIOMMU: Efficient IOMMU Emulation , 2011, USENIX Annual Technical Conference.

[44]  Mendel Rosenblum,et al.  It's Time for Low Latency , 2011, HotOS.

[45]  Information Networks Division,et al.  Netperf : A Network Performance Benchmark Revision 2.0 , 1995 .

[46]  Jamal Hadi Salim,et al.  Beyond Softnet , 2001, Annual Linux Showcase & Conference.

[47]  Mark Russinovich,et al.  Microsoft Windows Internals, Fourth Edition: Microsoft Windows Server(TM) 2003, Windows XP, and Windows 2000 (Pro-Developer) , 2004 .

[48]  Srilatha Manne,et al.  Accelerating two-dimensional page walks for virtualized systems , 2008, ASPLOS.

[49]  Muli Ben-Yehuda,et al.  Direct Device Assignment for Untrusted Fully-Virtualized Virtual Machines , 2008 .

[50]  Andy Oram,et al.  Understanding the Linux Kernel, Second Edition , 2002 .

[51]  Parameswaran Ramanathan,et al.  HIP: hybrid interrupt-polling for the network interface , 2001, OPSR.

[52]  Muli Ben-Yehuda,et al.  The Turtles Project: Design and Implementation of Nested Virtualization , 2010, OSDI.