The rise of eBPF for non-intrusive performance monitoring

In this paper, we explain that container engines are strengthening their isolation mechanisms. Therefore, non-intrusive monitoring becomes a must-have for the performance analysis of containerized user-space application in production environments. After a literature review and background of Linux subsystems and container isolation concepts, we present our lessons learned of using the extended Berkeley packet filter to monitor and profile performance. We carry out the profiling and tracing of several Interledger connectors using two full-fledged implementations of the Interledger protocol specifications.

[1]  Toke Høiland-Jørgensen,et al.  XDP-challenges and future work , 2018 .

[2]  Toke Høiland-Jørgensen,et al.  The eXpress data path: fast programmable packet processing in the operating system kernel , 2018, CoNEXT.

[3]  Luca Deri,et al.  Combining System Visibility and Security Using eBPF , 2019, ITASEC.

[4]  Marco Levorato,et al.  eBPF-based content and computation-aware communication for real-time edge computing , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[5]  Daniel Raumer,et al.  Performance Implications of Packet Filtering with Linux eBPF , 2018, 2018 30th International Teletraffic Congress (ITC 30).

[6]  Yong Zhao,et al.  vNetTracer: Efficient and Programmable Packet Tracing in Virtualized Networks , 2018, 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS).

[7]  Muhammad Hamad Alizai,et al.  InKeV: in-kernel distributed network virtualization for DCN , 2018, CCRV.

[8]  Jae-Hyoung Yoo,et al.  Design and Implementation of eBPF-based Virtual TAP for Inter-VM Traffic Monitoring , 2018, 2018 14th International Conference on Network and Service Management (CNSM).

[9]  Brendan Gregg.  Performance Superpowers with Enhanced {BPF} , 2017 .

[10]  Christopher B. Hauser,et al.  Reviewing Cloud Monitoring: Towards Cloud Resource Profiling , 2018, 2018 IEEE 11th International Conference on Cloud Computing (CLOUD).

[11]  Olivier Bonaventure,et al.  Making the Linux TCP stack more extensible with eBPF , 2019 .

[12]  Stefan Thomas,et al.  Interledger: Creating a Standard for Payments , 2016, WWW.

[13]  Jorge A. Navas,et al.  Simple and precise static analysis of untrusted Linux kernel extensions , 2019, PLDI.

[14]  Christina Delimitrou,et al.  X-Containers: Breaking Down Barriers to Improve Performance and Isolation of Cloud-Native Containers , 2019, ASPLOS.

[15]  Michael Wagner,et al.  Selective runtime monitoring: Non-intrusive elimination of high-frequency functions , 2014, 2014 International Conference on High Performance Computing & Simulation (HPCS).

[16]  Anima Anandkumar,et al.  Non-intrusive transaction monitoring using system logs , 2008, NOMS 2008 - 2008 IEEE Network Operations and Management Symposium.

[17]  JongWon Kim,et al.  Open-source IO visor eBPF-based packet tracing on multiple network interfaces of Linux boxes , 2017, 2017 International Conference on Information and Communication Technology Convergence (ICTC).

[18]  Konrad Slind,et al.  Monitoring distributed systems , 1987, TOCS.

[19]  Dirk Merkel,et al.  Docker: lightweight Linux containers for consistent development and deployment , 2014 .

[20]  Wenguang Chen,et al.  RACEZ: a lightweight and non-invasive race detection tool for production applications , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[21]  Carlo Emmanoel Tolla de Oliveira,et al.  A transparent and centralized performance management service for CORBA based applications , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[22]  Betsy Beyer,et al.  Monitoring Distributed Systems , 2016 .

[23]  Michel Dagenais,et al.  Survey and Analysis of Kernel and Userspace Tracers on Linux , 2018, ACM Comput. Surv..

[24]  Evan Schwartz.  A Payment Protocol of the Web, for the Web: Or, Finally Enabling Web Micropayments with the Interledger Protocol , 2016, WWW.

[25]  Dimitrios P. Pezaros,et al.  BPFabric: Data Plane Programmability for Software Defined Networks , 2017, 2017 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS).

[26]  Timothy Wood,et al.  Cloud-Scale Application Performance Monitoring with SDN and NFV , 2015, 2015 IEEE International Conference on Cloud Engineering.

[27]  Yuan He,et al.  An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems , 2019, ASPLOS.

[28]  Brendan Gregg.  BPF Performance Tools , 2019 .

[29]  Cheng-Zhong Xu,et al.  sMonitor: A Non-Intrusive Client-Perceived End-to-End Performance Monitor of Secured Internet Services , 2006, USENIX Annual Technical Conference, General Track.