Supporting experiments in computer systems research

Systems research is an experimental science. Most research in computer systems follows the trend of innovate (e.g. build a novel garbage collector) and then evaluate (e.g. does it significantly speed up our programs). Researchers use experiments to drive their work; they use experiments to identify bottlenecks and then again to determine if their innovations for addressing those bottlenecks are effective. If their experiments are not carried out properly, researchers may draw incorrect conclusions; they may end up wasting time on something that is not really a problem and may conclude their innovations are beneficial even when they are not. A complicating factor in computer systems experiments is the fact that computer systems are nonlinear dynamical systems, capable of complex and even chaotic behavior. A hallmark of chaos is a sensitive dependence on initial conditions—small changes to the system lead to a large effect on its overall behavior. This sensitivity complicates both observations of our systems and evaluations of our innovations. It complicates our observations because our measurement tools perturb the system they are observing. It complicates our evaluations because small changes to the environment in which we carry out our experiments can cause large and dramatic changes in system behavior. In this dissertation, we argue the systems community needs to support experiments with tools that allow a researcher to accurately observe her system and methodologies that allow researchers to accurately evaluate the impact of their innovations. To support our argument, we introduce two tools that allow researchers to accurately observe their application’s behavior and one methodology that allows researchers to accurately evaluate the impact of their innovations.