ArchExplorer.org: joint compiler/hardware exploration for fair comparison of architectures

While reproducing experimental results of research articles is standard practice in mature domains of science, such as physics or biology, it has not yet become mainstream in computer architecture. However, recent research shows that the lack of a fair and broad comparison of research ideas can be significantly detrimental to the progress, and thus the productivity, of research. At the same time, the complexity of architecture simulators and the fact that simulators are not systematically disseminated with novel ideas are largely responsible for this situation. While this methodology has a fundamental impact on research, it is by essence a practical issue. In this article, we present and put to task an atypical approach which aims at overcoming this practical methodology issue, and which takes the form of an open and continuous exploration through a server-side web infrastructure. First, rather than requiring from a researcher to engage in the daunting task of seeking, installing and running the simulators of many alternative mechanisms, we propose that researchers upload their simulator to the infrastructure, where the corresponding mechanism is automatically compared against all known ideas so far. Second, the comparison takes the form of a broad compiler/hardware exploration, so that a new mechanism is deemed superior only if it can outperform a tuned baseline and all known tuned mechanisms, for a given area and/or power budget. These two principles considerably facilitate a fair and quantitative comparison of research ideas. The web infrastructure is now publicly open, and we put the overall approach to task with a set of data cache mechanisms. We explain how the tools and methodological issues of contributed simulators can be overcome, and we show that this broad exploration can challenge some earlier assessments about data cache research.