Fine-grained dynamic instrumentation of commodity operating system kernels

We have developed a technology, fine-grained dynamic instrumentation of commodity kernels, which can splice (insert) dynamically generated code before almost any machine code instruction of a completely unmodified running commodity operating system kernel. This technology is well-suited to performance profiling, debugging, code coverage, security auditing, runtime code optimizations, and kernel extensions. We have designed and implemented a tool called Kernlnst that performs dynamic instrumentation on a stock production Solaris kernel running on an UltraSPARC, On top of KernInst, we have implemented a kernel performance profiling tool, and used it to understand kernel and application performance under a Web proxy server workload. We used this information to make two changes (one to the kernel, one to the proxy) that cumulatively reduce the percentage of elapsed time that the proxy spends opening disk cache files from 40% to 7%.

[1]  Margo I. Seltzer,et al.  Self-monitoring and self-adapting operating systems , 1997, Proceedings. The Sixth Workshop on Hot Topics in Operating Systems (Cat. No.97TB100133).

[2]  Lance M. Berc,et al.  Continuous profiling: where have all the cycles gone? , 1997, TOCS.

[3]  Richard J. Moore.  Dynamic Probes and Generalised Kernel Hooks Interface for Linux , 2000, Annual Linux Showcase & Conference.

[4]  David S. H. Rosenthal,et al.  Evolving the Vnode interface , 1990, USENIX Summer.

[5]  Calton Pu,et al.  Optimistic incremental specialization: streamlining a commercial operating system , 1995, SOSP.

[6]  Calton Pu,et al.  Fast concurrent dynamic linking for an adaptive operating system , 1996, Proceedings of International Conference on Configurable Distributed Systems.

[7]  Susan L. Graham,et al.  Gprof: A call graph execution profiler , 1982, SIGPLAN '82.

[8]  Larry L. Peterson,et al.  Making paths explicit in the Scout operating system , 1996, OSDI '96.

[9]  Calton Pu,et al.  A Study of Dynamic Optimization Techniques: Lessons and Directions in Kernel Design , 1993 .

[10]  Thomas Ball,et al.  Edge profiling versus path profiling: the showdown , 1998, POPL '98.

[11]  James R. Larus,et al.  Optimally profiling and tracing programs , 1992, POPL '92.

[12]  John H. Hartman,et al.  Joust: A Platform for Liquid Software , 1999, Computer.

[13]  Jeffrey C. Mogul,et al.  Scalable Kernel Performance for Internet Servers Under Realistic Loads , 1998, USENIX Annual Technical Conference.

[14]  Lance M. Berc,et al.  Continuous profiling: where have all the cycles gone? , 1997, ACM Trans. Comput. Syst..

[15]  Daeyeon Park,et al.  Improving the effectiveness of software prefetching with adaptive executions , 1996, Proceedings of the 1996 Conference on Parallel Architectures and Compilation Technique.

[16]  Barton P. Miller,et al.  Dynamic program instrumentation for scalable performance tools , 1994, Proceedings of IEEE Scalable High Performance Computing Conference.

[17]  George C. Necula,et al.  Safe kernel extensions without run-time checking , 1996, OSDI '96.

[18]  Steve R. Kleiman,et al.  Vnodes: An Architecture for Multiple File System Types in Sun UNIX , 1986, USENIX Summer.

[19]  Thomas E. Anderson,et al.  SLIC: An Extensibility System for Commodity Operating Systems , 1998, USENIX Annual Technical Conference.

[20]  Brian N. Bershad,et al.  Extensibility safety and performance in the SPIN operating system , 1995, SOSP.

[21]  Brian N. Bershad,et al.  Fast, effective dynamic compilation , 1996, PLDI '96.

[22]  Oscar Naim,et al.  Dynamic instrumentation of threaded applications , 1999, PPoPP '99.

[23]  Barton P. Miller,et al.  Dynamic control of performance monitoring on large scale parallel systems , 1993, ICS '93.

[24]  Zheng Wang,et al.  System support for automatic profiling and optimization , 1997, SOSP.

[25]  Robert J. Fowler,et al.  Adaptive cache coherency for detecting migratory shared data , 1993, ISCA '93.

[26]  Calton Pu,et al.  Threads and input/output in the synthesis kernal , 1989, SOSP '89.

[27]  Michael D. Smith,et al.  Tracing with Pixie , 1991 .

[28]  Peter Lee,et al.  Optimizing ML with run-time code generation , 1996, PLDI '96.

[29]  James R. Larus,et al.  Rewriting executable files to measure program behavior , 1994, Softw. Pract. Exp..

[30]  Calton Pu,et al.  The Synthesis Kernel , 1988, Comput. Syst..

[31]  George C. Necula,et al.  Proof-carrying code , 1997, POPL '97.

[32]  Uresh K. Vahalia.  UNIX Internals: The New Frontiers , 1995 .

[33]  Geoffrey H. Kuenning.  Kitrace: Precise interactive measurement of operating systems kernels , 1995, Softw. Pract. Exp..

[34]  Barton P. Miller,et al.  Using Dynamic Kernel Instrumentation for Kernel and Application Tuning , 1999, Int. J. High Perform. Comput. Appl..

[35]  Christopher Small.  MiSFIT: A Tool for Constructing Safe Extensible C++ Systems , 1997, COOTS.

[36]  Barton P. Miller,et al.  Typestate Checking of Machine Code , 2001, ESOP.

[37]  P. Geoffrey Lowney,et al.  Feedback directed optimization in Compaq's compilation tools for Alpha , 1999 .

[38]  Robert Wahbe,et al.  Efficient software-based fault isolation , 1994, SOSP '93.

[39]  Peter Deutsch,et al.  A Flexible Measurement Tool for Software Systems , 1971, IFIP Congress.

[40]  Adrian Cockcroft,et al.  Sun Performance and Tuning: Java and the Internet , 1998 .

[41]  L PetersonLarry,et al.  Analysis of techniques to improve protocol processing latency , 1996 .

[42]  Richard McDougall,et al.  Solaris internals : core kernel components , 2001 .

[43]  Brian N. Bershad,et al.  Safe Dynamic Linking in an Extensible Operating System , 1999 .

[44]  Martin C. Rinard,et al.  Dynamic feedback: an effective technique for adaptive computing , 1997, PLDI '97.

[45]  Michael D. Smith,et al.  Overcoming the Challenges to Feedback-Directed Optimization , 2000, Dynamo.

[46]  Robert Grimm,et al.  Application performance and flexibility on exokernel systems , 1997, SOSP.

[47]  Brian N. Bershad,et al.  Dynamic binding for an extensible system , 1996, OSDI '96.

[48]  Larry L. Peterson,et al.  Analysis of techniques to improve protocol processing latency , 1996, SIGCOMM '96.

[49]  Donald E. Knuth,et al.  Optimal measurement points for program frequency counts , 1973 .

[50]  Karl Pettis,et al.  Profile guided code positioning , 1990, PLDI '90.

[51]  Sorin Lerner,et al.  Mojo: A Dynamic Optimization System , 2000 .

[52]  Dawson R. Engler,et al.  VCODE: a retargetable, extensible, very fast dynamic code generation system , 1996, PLDI '96.

[53]  Carl Ponder,et al.  Inaccuracies in program profilers , 1988, Softw. Pract. Exp..

[54]  Barton P. Miller,et al.  A Callgraph-Based Search Strategy for Automated Performance Diagnosis (Distinguished Paper) , 2000, Euro-Par.

[55]  David R. Cheriton,et al.  A caching model of operating system kernel functionality , 1994, OSDI '94.

[56]  Dawson R. Engler,et al.  Exokernel: an operating system architecture for application-level resource management , 1995, SOSP.

[57]  Oscar Naim,et al.  MDL: a language and compiler for dynamic program instrumentation , 1997, Proceedings 1997 International Conference on Parallel Architectures and Compilation Techniques.

[58]  Larry L. Peterson,et al.  Scout: a communications-oriented operating system , 1995, Proceedings 5th Workshop on Hot Topics in Operating Systems (HotOS-V).

[59]  Robert L. Probert,et al.  Optimal Insertion of Software Probes in Well-Delimited Programs , 1982, IEEE Transactions on Software Engineering.

[60]  Margo I. Seltzer,et al.  Dealing with disaster: surviving misbehaved kernel extensions , 1996, OSDI '96.

[61]  Calton Pu,et al.  Specialization classes: an object framework for specialization , 1996, Proceedings of the Fifth International Workshop on Object-Orientation in Operation Systems.

[62]  Dawson R. Engler,et al.  DPF: fast, flexible message demultiplexing using dynamic code generation , 1996, SIGCOMM 1996.

[63]  Alec Wolman,et al.  Instrumentation and optimization of Win32/intel executables using Etch , 1997 .

[64]  Barton P. Miller,et al.  Safety checking of machine code , 2000, PLDI '00.

[65]  Jeffrey Dean,et al.  ProfileMe: hardware support for instruction-level profiling on out-of-order processors , 1997, Proceedings of 30th Annual International Symposium on Microarchitecture.

[66]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[67]  David R. Kaeli,et al.  Tracing and Characterization of Windows NT-based System Workloads , 1998, Digit. Tech. J..

[68]  George C. Necula,et al.  The design and implementation of a certifying compiler , 1998, PLDI.

[69]  Calton Pu,et al.  Microlanguages for Operating System Specialization , 1997 .

[70]  Dawson R. Engler,et al.  The Operating System Kernel as a Secure Programmable Machine , 1994, ACM SIGOPS European Workshop.

[71]  Brian N. Bershad,et al.  An Extensible Protocol Architecture for Application-Specific Networking , 1996, USENIX Annual Technical Conference.

[72]  Butler W. Lampson.  Interactive machine language programming , 1965 .

[73]  James R. Larus,et al.  EEL: machine-independent executable editing , 1995, PLDI '95.