Dynamic Probes and Generalised Kernel Hooks Interface for Linux

Dynamic Probes (Dprobes)[1] is a generic and pervasive system debugging facility that will operate under the most extreme software conditions such as debugging a deep rooted operating system problems in a live environment. For example, page-manager bugs in the kernel or perhaps user or system problems that will not re-create easily in either a lab or production environment. For such inaccessible problem scenarios Dprobes not only offers a technique for gathering diagnostic information but has a high probability of successful outcome without the need to build custom modules for debugging purposes. Dprobes allows the insertion of fully automated breakpoints or probepoints, anywhere in the system and user space. Probepoints are global by definition, that is they are defined relative to a module and not to a storage address. Each probepoint has an associated set of probe instructions that are interpreted when the probe fires. These instructions allow memory and CPU registers to be examined and altered using conditional logic. When the probe program terminates an external debugging facility may be optionally triggered - should it register for this purpose. For example: A trace facility may augment its capability with a dynamic trace capability by using the Dprobes facility as a means of inserting tracepoints - dynamically, without any prior code modification. A crash dump facility may use Dprobes as a means of invoking dumps conditionally when a specific set of circumstances occurs in a particular code path. A debugger may use Dprobes as high-speed complex conditional breakpoint service. This paper describes the architecture of Dynamic Probes and briefly discusses a couple of examples of its successful application. In creating Dynamic Probes, we were challenged with the conflicts between: Size of the kernel modification Co-existence with other kernel enhancements, particularly debugging and instrumentation enhancements. Maintaining concurrency with the latest kernel version. Ease of development and continued enhancement of Dynamic Probes. We alleviated these problems by developing a generalised interface for kernel modifications to use allowing them to exist as dynamically loadable kernel modules. This interface: The Generalised Kernel Hooks Interface (GKHI) is described in the second part of this paper.